Introduction
Measuring social impact often falls into the trap of a technocratic fetishization of numbers. The true effectiveness of an organization stems not from the accumulation of data, but from the ability to interpret that data and implement strategic corrections.
In this article, we explore Matt Stevenson-Dodd's model, which proposes a shift from retrospective evaluation toward a Live Data system. You will learn how to measure intangible phenomena and why technology must support human agency rather than replace it.
Measuring Latent Constructs Requires Theoretical Rigor
Phenomena such as dignity or well-being are latent constructs. They possess no natural unit of measurement, meaning they cannot be measured with a simple "thermometer." Stevenson-Dodd argues that reliable measurement requires psychometric and theoretical rigor.
Instead of proprietary surveys, the author suggests using validated tools, such as the Warwick-Edinburgh Mental Well-being Scale (WEMWBS). This avoids subjectivism and ensures data comparability without sacrificing the depth of lived experience.
It is crucial to understand that a scale score is not well-being itself, but rather the output of an instrument based on a specific theoretical model. What we measure depends entirely on how we have previously defined the essence of the phenomenon under study.
The Pitfalls of Averages and the Necessity of Effect Distribution Analysis
Relying on an average score is risky, as the mean can be anthropologically deceptive. It may mask the fact that a program helps those already in a better position, while the most vulnerable group experiences a regression.
Stevenson-Dodd emphasizes the importance of analyzing heterogeneous treatment effects. An organization must ask not only whether an intervention works on average, but specifically for whom it is effective and for whom it may actually be harmful.
Social impact is primarily about the distribution of effects. Ignoring the distribution of results leads to an illusion of success and rewards entities that work with "easier" cases, which contradicts the core social mission.
Qualitative Data as a Rigorous Evidentiary System
Qualitative data is not merely a supplement to statistics. It serves as a rigorous evidentiary system that answers "how?" and "why?", explaining mechanisms of change that are inaccessible to numbers.
The author contrasts simple anecdotes with the combination of Data + Story. Quantitative data provides the evidentiary skeleton, while narrative provides the context. Triangulating both sources allows for the detection of contradictions—for example, when participants report satisfaction with a program despite a lack of sustainable results.
In this process, AI can support analysis through transcription or coding, but it cannot replace the researcher. Interpreting meaning, irony, or power dynamics requires human reflexivity and scaled hermeneutics.
Conclusion
Organizational maturity is manifested in the transition from monitoring to double-loop learning. This means not only adjusting actions to achieve a goal, but having the courage to question the underlying assumptions and definitions of success in the face of evidence.
The highest form of development is not possessing the most accurate dashboard, but cultivating an epistemology of humility. True success is measured by the speed with which a system can discover where it was mistaken and how to adapt its identity to the real-world experiences of people.
Frequently Asked Questions
How can intangible phenomena, such as well-being or dignity, be measured reliably?
Intangible phenomena are measured indirectly, inferring their intensity based on responses, behaviors, and relationships using tools from psychometrics and qualitative research. Reliable measurement requires the use of validated instruments (e.g., the WEMWBS scale) that are aligned with the adopted theory of the phenomenon being studied and demonstrate sensitivity to actual change.
1. Why is relying on the average score of a scale insufficient and risky when measuring social impact?
2. Relying on the average score is risky because the mean can mask the fact that a program fails to reach those in the most difficult situations or harms specific subgroups. Social impact depends on the distribution of effects, not just average efficiency, as the "average participant" is merely a statistical construct that does not reflect real differences between people.
3. Are qualitative data less reliable than quantitative data, and what role do they play in the rigorous measurement of social impact?
4. Qualitative data are not less reliable than quantitative data; they differ only in how they reduce reality and can be methodologically robust through rigors such as triangulation or theoretical saturation. In measuring social impact, they play a key role by explaining the mechanisms of change (answering "how?" and "why?") and allowing for the identification of both intended and unintended effects of actions.
5. Why might traditional surveys and administrative data be insufficient or misleading in measuring social impact?
6. Surveys can be misleading due to the phenomenon of response shift, where an increase in a beneficiary's aspirations leads to a lower self-assessment despite a real improvement in their situation. Additionally, attrition bias occurs—people in the most difficult situations are more likely to drop out of the study, which artificially inflates the results. Meanwhile, administrative data are insufficient because they only record institutional facts, ignoring subjective human experiences and feelings.
7. Can artificial intelligence fully replace a researcher in the qualitative analysis of social data?
8. Artificial intelligence cannot fully replace a researcher because qualitative analysis relies on the interpretation of meaning and social context, which AI does not understand. This technology should serve as support (human-AI triangulation), assisting in data processing and hypothesis falsification, while the responsibility for final interpretation remains with the human.
9. What ethical and methodological risks are associated with using AI and secondary data in social impact analysis?
10. The main risks include AI replicating dominant patterns and pathologizing descriptions, as well as the risk of misallocating resources resulting from an uneven distribution of coding errors, particularly among marginalized groups. Ethical challenges include difficulties in fully anonymizing rich qualitative narratives and the issue of informational legitimacy for individuals whose data are processed automatically. Additionally, when using secondary data, there is a risk of construct misinterpretation, where statistical correlations may not reflect actual social phenomena.
How should impact measurement methods be selected to avoid the trap of relying solely on data and reliably capture the complexity of social changes?
A proportional research framework should be chosen, where the level of complexity corresponds to the decision risk and the social significance of the program, combining quantitative and qualitative methods. In the case of advocacy, it is worth using tools such as outcome harvesting and process tracing, while incorporating the lived experience of those directly affected by the change during the interpretation of results.
How can technology and real-time data transform an organization from a bureaucratic reporting machine into an adaptive system?
Technology and real-time data transform an organization by shifting the function of information from documenting the past to creating a feedback loop between action and decision. Instead of relying on retrospective reports, Live Data systems allow for the continuous observation of discrepancies between the desired and actual state, enabling faster modification or termination of ineffective actions.
Why does simply possessing up-to-date real-time data not guarantee effective social impact management?
Possessing real-time data alone does not guarantee effective impact management because, without appropriate institutional mechanisms and organizational culture, it may only lead to a faster optimization of flawed assumptions. Effectiveness requires a transition from simple operational corrections (single-loop learning) to questioning the goals and theories of change themselves (double-loop learning), as well as reflection on the organization's learning processes.
Why does access to real-time data not always mean the ability to effectively and quickly steer social impact?
Reacting quickly to short-term data fluctuations can lead to oversteering the system based on noise rather than signal, which increases its variance. Furthermore, organizations may possess real-time data but struggle with decision-making delays or high observability while lacking actual tools to control the given process.
How can real-time data be transformed into a tool for genuine organizational steering and strategy updates?
A Live Data Storytelling system should be implemented, based on the distinction between outcome indicators, context, and leading indicators that serve as prognostic signals. It is crucial to ensure causal interpretability through a drill-down function, allowing a transition from a general result to a specific diagnosis and hypothesis. Organizational steering should be based on adaptive loops and methodological discipline, where strategy corrections are justified by evidence rather than impulsive reactions.
Why does Live Data technology alone not guarantee that an organization will actually learn from data?
Live Data technology reduces information lags but does not automatically remove psychological barriers related to accepting information that contradicts the organization's identity or interests. Without psychological safety, data may be delayed, filtered, or misinterpreted, and the fear of sanctions for failure leads to hiding problems instead of authentic learning.
What does it mean in practice to be a data-driven organization, and how can one distinguish technocratic monitoring from actual organizational learning?
Being a data-driven organization in practice means having a short and honest path from information to specific corrections in actions, decisions, or strategy. Actual organizational learning is distinguished from technocratic monitoring by the ability to revise goals and theories of change (double-loop governance) and reflection on how knowledge is created, rather than merely mechanically tracking indicators.
