Measuring impact: lessons learned from investing in media

By Magdalena Skrzypek, Impact and Communications Manager

Measuring the impact of media and the impact of media support has long been a major challenge for our sector. How do media development organisations assess whether and to what degree their work has had an impact? For MDIF, we need to know that our financing and business support are making a difference. Are we achieving our purposes as agents of change? How are media companies we invest in transforming their communities?

At MDIF, impact assessment is a critical part of our work. Since 2005, we have published our Media Development Impact Dashboard to publicly present the findings of our annual analysis. Among other findings, our most recent report published this week revealed that clients increase their revenues by 205% and reach by 190% on average after five years of working with MDIF, and that 95% of MDIF-supported media report carrying out work around Covid-19 that had a demonstrable impact on their communities in 2020.

Whether media support organisations are looking at the direct impact of their work on media outlets or at media’s impacts on their societies, methodologies can quickly become complex. As we continue to wrestle with methodological challenges and strive to balance simplicity and meaningful findings, here we reflect on our 15 years of experience of measuring and reporting social value. Although closely tied to our expertise – independent media – we believe our lessons learned can also offer insights to other niche sectors approaching impact measurement.

Develop a system that works for you

At MDIF, we focus our impact assessment efforts on two areas. First, we track how MDIF-supported organisations change over the course of their involvement with us in terms of their reach, revenues and viability. We also hear directly from supported media organisations about the changes their businesses have experienced, including their evaluation of the extent to which MDIF financing and business support has impacted on their companies. Second, to monitor how our investees create impact in their communities, we focus on their journalism and information-sharing around topics of corruption and accountability, elections, social issues (like the environment, gender, minorities, immigration or LGBTQ+) and, specifically this year, Covid-19.

This custom approach is tailored to our mission to support independent media around the world providing the news, information and debate that people need to build free, thriving societies. It has been developed over the years and evolves with each annual Dashboard published. While some of the focus areas and data points date back to 1999 when we began to collect client data in a standardised format, we continuously strive to refine our approach in response to sweeping changes in the media sector and changing best practices in impact measurement, recognising that for any impact investor, the impact measurement should be seen as a learning process and continuous improvements are necessary.

Embed impact measurement in the investment process

The approach to impact measurement should reflect the mission and activities of the organisation and be closely aligned to the investment strategy and operations. Making impact measurement integral to the investment process facilitates the end goal of maximising impact. Let’s take additionality. MDIF provides affordable debt and equity financing with no editorial strings attached to independent media businesses in challenging environments. We make sure that our investments are additional to the activities of the commercial market, in that they are extended to media businesses who could not obtain adequate financing from commercial sources on an economically feasible basis, or those that could do so would be subjected to conditions that would threaten their editorial independence. These conditions for our financing are laid out in our eligibility criteria. Through the lens of additionality, an investment from one of our funds has an immediate benefit to the media companies we back, enabling them to go on to create positive impact for their communities.

Consider tying profit participation of the fund manager to the social impact of investments, as we did with our Emerging Media Opportunity Fund (EMOF), the first private equity fund for independent media in countries where access to balanced news and information is under threat. To ensure financial and mission alignment, we tied 100% of our profit participation to the social impact of investments: MDIF’s earnings through carried interest are conditioned on meeting a range of mission-aligned indicators.

Identify the right information and leverage multiple data sources

To get the most accurate picture of our and our clients’ impact, we combine various data sources, including data readily available to us through clients’ quarterly monitoring and an annual Impact Dashboard survey, as well as from several external data sources. By combining data from different sources, we can paint a more insightful picture. For instance, to monitor the online reach of our clients, we rely on data gathered by Google Analytics. To add context across different impact areas, we also quantify survey responses against comparable and pertinent indicators, such as the World Press Freedom Index published by Reporters Without Borders, the Social Progress Index by Social Progress Imperative and, this year, the Lowy Institute’s Covid Performance Index.

Our selection of impact priorities developed in-house, although based on existing research and theory, does not aim to be comprehensive, but rather illuminates, by collecting concrete examples of our work and the work of our clients. While collecting too little information jeopardises the ability to tell story of impact, over-collecting data makes data processing and storing more complex. And think carefully about what you asking of the organisations you work with. We found that with our survey, too many questions overwhelm and tire the respondents, resulting in increased drop-off rates. We try to avoid excessive data collection, instead prioritising data that ultimately delivers useful investment information – for example, adding context to performance trends due to Covid-19, when it helped us to determine how we prioritise our planning and technical assistance. When collecting impact data we try to follow a core principle of prioritising efficiency and understanding the day-to-day business realities of running the media companies we support.

Numbers don’t tell the whole story

We present a range of metrics in the Impact Dashboard, from changes in revenues to changes in investees’ reach. The metrics we have chosen are based on their relevancy for the largest number of MDIF-supported organisations and to achieve a standard measurement across our portfolio. Yet, we believe that numbers alone are insufficient to unearth or communicate impact, especially when describing how media impact on their communities.

From changing lives to changing laws, media produce many forms of public benefit that make our governments more transparent and less corrupt, and our societies more informed and inclusive. To monitor how MDIF-supported media create impact that brings about transformative changes, we show examples of their powerful and impactful journalism and information-sharing, and describe the real-world changes arising from it, whether it is the start of a citizen-lead protest, the enactment of a new law, or the dismissal of an incompetent official. Through stories of change, we seek to convey the societal value of the journalistic work of media companies we support and, on a broader level, inform the public about why journalism matters and why it has critical civic importance.

Be transparent and don’t overclaim

To mitigate the risk of ‘impact washing’, we seek to be transparent and accountable when discussing the effects of our investments. We acknowledge that the data we collect has its limitations and that the absence of a relevant control group means that we are unable to assess displacement, deadweight, drop off and attribute impact to a particular intervention. We make it clear that we view our investment as a contributor to, not the sole cause of, our clients’ growth, and use the collected data to monitor the companies we support and make more informed decisions around our portfolio. Similarly, we are also very careful not to attribute causality and we view our clients’ work as a contributor to, not the sole cause of, the changes that occurred in their communities.

Although we grapple with different challenges in both collecting and standardising data across our diverse portfolio, from complicated causality to uncertain audience research data in many emerging markets, our objective is to collect data with an appropriate degree of rigour that allows us to provide an accurate and reliable insight into our work. We take due diligence and care in reducing inaccuracies and errors to the extent possible, validating the results clients report, cross-referencing them with our data and third-party services, and eliminating or adjusting anomalous figures. As we continue to address these and other challenges, we believe that full transparency regarding our methodology is important both for accountability and learning (see the full Impact Dashboard Methodology for more detailed information).

Alignment to existing social impact frameworks and SDGs

In recent years, there has been an emergence of standardised metrics to measure social impact. So far, the standards are very broad and, due to the specificity of our investments, their one-size-fits-all approach does not adequately meet our needs, nor are they aligned with our mission and goals. Industry-specific metrics have been developed for evaluating work in areas such as banking, health care and the environment, but not for media. Given the still-evolving status of impact measurement and management practices, we are constantly looking to improve our approach and closely follow developments as they happen.

Many impact investors have been also orienting themselves towards the UN Sustainable Development Goals (SDGs). We see it as a positive development and are also exploring closer alignment. As the struggle to achieve Agenda 2030 takes on ever greater urgency, the role of independent media in achieving the SDGs is crucial. Not only is media specifically included in SDG 16 (public access to information is part of Target 16.10), they support the attainment of other SDGs because reliable, pluralistic information underpins other development targets: from SDG 2 No Hunger (as Nobel economics laureate Amartya Sen stated, no country with a free press has ever experienced famine) to SDG 5 Gender Equality and SDG 10 Reduced Inequalities, independent media give society a platform to express their voices and hold authorities to account.

Our continuing challenge is to demonstrate these complex connections in a reliable, transparent and comprehensive way.


This article is part of our series, ‘25 things we’ve learned’, marking MDIF’s 25th anniversary.

Share article