Our approach

Measuring the impact of media support has long been a major challenge across the sector. However, recognising how valuable a reliable measurement of our impact would be, we track the performance of each client throughout the length of our relationship, enabling us to assess our impact on their business over time. This measurement helps us better understand the results of our work and allows us to make more informed decisions around the performance of our portfolio. Additionally, we also track the impact of our clients upon society to better convey the societal value of their journalistic work. On a broader level, we simply want to inform the public about why journalism matters and why it has critical civic importance. Overall, by understanding this very real return on investment – the social and financial results of media financing – MDIF, its donors and the development community may better support independent news businesses.

We focus our impact assessment efforts on two areas: first, direct impact of our investment on clients; and, second, our clients’ impacts on their societies. The methodology is designed to assess established media outlets that produce news and information content and are focused on generating revenues to support their operations, rather than early-stage digital startups or businesses that provide platforms to connect users instead of providing news and information to audiences. While the outlets we invest in are diverse in terms of their geographic focus, revenue models and media type, we believe that these metrics are relevant for the largest number of clients in our portfolio.

To assess the extent to which our support contributes to our clients’ long-term sustainability, we evaluate how each media company’s revenues, reach and viability change over the course of their involvement with MDIF. Revenue data, which refers to combined company revenues from all services, including circulation, advertising and printing, is readily available through monthly reports from clients. To assess reach, MDIF encourages clients to use industry standards when reporting data, including using multipliers for print reach, conducting audience research on broadcast clients and using Google Analytics to assess online audiences. Financial viability, which determines the overall sustainability of client businesses by using an MDIF-generated risk-rating scale, is re-assessed three times each year by MDIF’s management and the metrics are evaluated annually by independent auditors to ensure the validity of the process. We also present client evaluation of impact: based on survey results, we hear directly from supported media organisations about the changes their businesses have experienced, including their evaluation of the extent to which MDIF financing and technical assistance has impacted on their companies.

To assess the extent to which the independent media supported by MDIF impact on their societies, we present powerful journalistic stories produced by clients that curb corruption and foster accountability. We also present our clients’ efforts to provide reliable information and shine a light on issues that impact people’s lives but are often left in the shadows, like the environment, gender, minorities, immigration and LGBT. To show how our clients encourage democratic participation, we highlight examples of their election reporting. These areas have been selected based on empirical research, including studies on media influence and media affects, but also our first-hand experience. When presenting these concrete examples we go beyond the metrics of audience reach and website traffic and try to focus on the ultimate social outcomes that followed, for example investigations conducted, laws changed or protests staged. Still, we are very careful not to attribute causality – in most cases we view our clients’ work as a contributor to, not the sole cause of, changes in communities.

MDIF gathers the data directly from our clients through monitoring and questionnaires. At the beginning of each calendar year, we approach MDIF-supported media with a survey that collects information on their reach, as well as corruption, accountability, election and social issues reporting for the preceding year. To the extent possible, we validate the reach results clients report, eliminating or adjusting anomalous figures. We also verify the stories by reviewing media coverage of those events, both by our client and other media outlets in the market. For a small number of clients that are unable to complete the survey, we evaluate a sample of their content from the previous year to identifying instances of high-impact reporting. Revenues data and financial viability, composed of seven company-specific indicators, are collected and updated regularly, with the final assessment for the year used for the annual analysis.

In addition to client records, we ingest data from several external data sources. For instance, to monitor the online reach of our clients, we rely on data gathered by Google Analytics. Across different impact areas, we quantify survey responses against comparable and pertinent indicators, including the World Press Freedom Index published by Reporters Without Borders, Corruption Perceptions Index by Transparency International, Social Progress Index by Social Progress Imperative, as well as the World Bank’s Political Stability and Absence of Violence Index and Ease of Doing Business Index.

Since 2005, we have published our Media Developoment Impact Dashboard to publicly present the findings of our annual analysis. The Dashboard provides a comprehensive description of impact results from the preceding year, as well as a longitudinal analysis of outcomes across our portfolio, including quantitative data on sales, reach and financial viability, in order to assess our clients’ economic sustainability, as well as qualitative data on our clients’ impact on society. Additionally, we present our impact in a form of case studies published on our website.

Overall, our impact measurements follow a core principle of prioritising efficiency and reflect the day-to-day business realities of media companies we support. We acknowledge that the data we collect has its limitations and that the absence of a relevant control group means that we are unable to attribute impact to a particular intervention. Although we grapple with issues in both collecting and standardising data across our diverse portfolio, from complicated causality to unreliability of audience research data in many emerging markets, our objective is to collect data with an appropriate degree of rigour that allows us to provide an accurate and reliable insight into our work. Given the still-evolving status of tracking impact and the sweeping changes in the media sector, we are constantly learning and striving to improve our approach. For a complete description of the challenges and how we try to address them see the full Impact Dashboard Methodology.