In the world of impact measurement, accurately attributing outcomes to specific interventions is crucial. This is where the concept of counterfactuals comes into play, ensuring that we give credit for tangible, actual changes that can demonstrably be accredited to your product or service alone. This blog post will explore what counterfactuals are and how to model your impact using them.
Counterfactuals are hypothetical scenarios that represent what would have happened in the absence of a particular intervention or action. In impact measurement, they are crucial for determining whether observed outcomes can be genuinely attributed to an intervention or if they would have occurred anyway. By comparing actual outcomes with these counterfactual scenarios, organizations can isolate and quantify the true impact of their interventions. This process ensures accurate attribution, prevents overstatement of results, and provides credible evidence of impact, which builds trust with stakeholders such as investors, customers, and partners who need assurance that their contributions are making a real difference.
The importance of counterfactuals extends beyond mere attribution; they enable organizations to allocate resources more effectively towards initiatives that generate the most significant positive outcomes that are uniquely possible through their innovation. By identifying which interventions are most effective, organizations can refine their strategies over time, optimizing their impact and developing a compelling narrative around their efforts. This data-driven approach enhances the accuracy of impact reporting and supports informed decision-making, allowing organizations to focus their efforts on maximizing social and environmental benefits. In today’s impact-focused landscape, counterfactuals serve as the foundation for rigorous impact measurement, ensuring that organizations give credit only for tangible, actual changes that can be demonstrably accredited to their products or services.
At Impactable, we've integrated a robust counterfactual analysis into our impact measurement framework to ensure accurate and reliable impact attribution. Here's how our process works:
We begin by establishing baseline scenarios that represent the state of the world without your intervention. This crucial step involves:
- Drawing upon existing research to suggest the most common pre-intervention situations
- Allowing you to propose additional scenarios specific to your context
- Collecting key indicator data before implementing your product or service
Next, we determine the frequency at which each baseline scenario applies to your specific metric:
- Impactable suggests percentages based on relevant research
- You can adjust these percentages using proprietary data or insights about your unique customer base
- We collaborate with you to develop ongoing data collection strategies, such as customer surveys, to refine these estimates over time
Finally, we quantify your intervention's genuine impact:
- Identify the social and environmental outcomes resulting from your metric
- Compare these outcomes to those of the baseline (counterfactual) scenarios
- Highlight the differences attributable to your product or service
Let's consider a company that places returning citizens into full-time truck driver jobs. To accurately measure their impact, we must consider three relevant counterfactual scenarios:
1. Returning citizens would otherwise be unemployed (jobless and actively seeking work)
2. Returning citizens would be otherwise jobless and not seeking work (not in labor force)
3. Returning citizens would be otherwise employed but would earn less
For each scenario, we determine the percentage of placed individuals that would fall into each category. At Impactable, we provide data to help estimate these percentages, but we also work with companies to develop custom data collection methods, such as surveying their customer base.
Let's break down the impact attribution for each counterfactual:
1. Unemployed (27%): For this group, we credit outcomes such as increased income (compared to $0 previously earned), decreased recidivism rates associated with employment, reduced people on unemployment and therefore government spending on unemployment benefits.
2. Not in Labor Force (36%): This group experiences similar outcomes to the first counterfactual, except for the reduction in unemployment benefits, as they were considered out of the workforce and would not have been receiving these benefits.
3. Employed at lower wages (<37%): Research indicates these individuals earn an average of $47K less than truck drivers. While we don't credit reduced unemployment or recidivism prevention for this group, we do account for the significant increase in income for these marginalized individuals.
This example illustrates the importance of considering all relevant counterfactuals when measuring impact. By analyzing these alternative scenarios, we can accurately attribute outcomes to the intervention, providing a comprehensive and nuanced understanding of the true impact created.
When discussing impact, it is vital to give credit for tangible, actual changes. If the same outcomes would have been reached without your product or service, then it is not truly making a difference. By incorporating counterfactual analysis, we ensure that the impact attributed to an intervention is genuine and not inflated by external factors.
If you’d like to learn more about how to accurately measure your attributable impact, book a call with Impactable, the sole impact measurement platform to incorporate counterfactuals into impact models!
Main image source: https://www.semanticscholar.org/paper/Impact-Evaluation-Methods-in-Public-Finance-Pomeranz/6bb4449af65a0545a979053a6a99f9e24a61c615
Find out how Impactable can help your organization scale.
REQUEST A DEMO
