Impact assessment is a critical part of our work, and we continuously work to refine and improve our approach in response to sweeping changes in the media sector and changing best practices in impact measurement. We have developed a streamlined process of tracking the impact of our and our clients’ work, combining numerical and narrative data.
Why do we track impact?
Measuring the impact of media support has long been a major challenge across the sector. However, recognizing how valuable a reliable measurement of our impact would be, we track the performance of each client throughout the length of our relationship, enabling us to assess our impact on their business over time. This measurement helps us better understand the results of our work and allows us to make more informed decisions around the performance of our portfolio. Additionally, we also track the impact of our clients upon society to better convey the societal value of their journalistic work. On a broader level, we simply want to inform the public about why journalism matters and why it has critical civic importance. Overall, by understanding this very real return on investment – the social and financial results of media financing – MDIF, its donors and the development community may better support independent news businesses.
What is the focus of our impact tracking?
We focus our impact assessment efforts on two areas: first, direct impact of our investment on clients; and, second, our clients’ impacts on their societies. The methodology is designed to assess established media outlets that produce news and information content and are focused on generating revenues to support their operations, rather than early-stage digital startups or businesses that provide platforms to connect users instead of providing news and information to audiences. While the outlets we invest in are diverse in terms of their geographic focus, revenue models, and media type, we believe that these metrics are relevant for the largest number of clients in our portfolio.
How do we track impact of our investment on clients?
To assess the extent to which our support contributes to our clients’ long-term sustainability, we evaluate how each media company’s sales, reach and viability change over the course of their involvement with MDIF. Sales data, which refers to combined company’s revenues from all services, including circulation, advertising and printing, is readily available through monthly reports from clients. To assess reach, MDIF encourages clients to use industry standards when reporting data, including using multipliers for print reach, conducting audience research on broadcast clients, and using Google Analytics to assess online audiences. Financial viability, which determines the overall sustainability of client businesses by using an MDIF-generated risk-rating scale, is re-assessed three times each year by MDIF’s management and the metrics are evaluated annually by independent auditors to ensure the validity of the process.
How do we track our clients’ impacts on their societies?
To assess the extent to which the independent media supported by MDIF impact on their societies, we present powerful journalistic stories produced by clients that curb corruption and foster accountability. We also present our clients’ efforts to provide reliable information and shine a light on issues that impact people’s lives but are often left in the shadows, like the environment, gender, ethnicity and LGBT. To show how our clients encourage democratic participation, we highlight examples of their election reporting. These areas have been selected based on empirical research, including studies on media influence and media affects, but also our first-hand experience. When presenting these concrete examples we go beyond the metrics of audience reach and website traffic and try to focus on the ultimate social outcomes that followed, for example investigations conducted, laws changed or protests staged. Still, we are very careful not to attribute causality – in most cases we view our clients’ work as a contributor to, not the sole cause of, changes in communities.
How do we collect data?
MDIF gathers the data directly from our clients through monitoring and questionnaires. At the beginning of each calendar year, we approach MDIF-supported media with a survey that collects information on their reach, as well as corruption, accountability, election and social issues reporting for the preceding year. To the extent possible, we validate the reach results clients report, eliminating or adjusting anomalous figures. For online reach, we rely on Google Analytics data. We also verify the stories by reviewing media coverage of those events, both by our client and other media outlets in the market. For a small number of clients that are unable to complete the survey, we evaluate a sample of their content from the previous year to identifying instances of high-impact reporting. Sales data and financial viability, composed of seven company-specific indicators, are collected and updated regularly, with the final assessment for the year used for the annual analysis.
How do we present our impact?
Since 2005, we have published our Media Developoment Impact Dashboard to publicly present the findings of our annual analysis. The Dashboard provides a comprehensive description of impact results from the preceding year, as well as a longitudinal analysis of outcomes across our portfolio, including quantitative data on sales, reach and financial viability, in order to assess our clients’ economic sustainability, as well as qualitative data on our clients’ impact on society. Additionally, we present our impact in a form of case studies published on our website.
What are the challenges we face?
We are facing multiple challenges when it comes to impact tracking, including complicated causality, double counting of reach for outlets that distribute news both online and offline, and unreliability of audience research data in many emerging markets. Although we grapple with issues in both collecting and standardizing data across our diverse portfolio, we make every effort to ensure that we provide an accurate and reliable insight into our work. Given the still-evolving status of tracking impact, we’re constantly learning and improving our approach. As we continue to address challenges we believe that full transparency regarding our methodology is important both for accountability and learning. For a complete description of the challenges and how we try to address them see the full Impact Dashboard Methodology.