Three weeks ago, I began to share a number of sparks of engagement and reflections that have been stimulated by my recent explorations around complexity, relationships and collaborative learning, for social change.
I’m sharing these as bite-sized pieces, with my sparks of engagement reformulated as learning questions. For each question, I’ve included links to the initial spark of engagement, and then shared my subsequent reflections.
Having set the scene, I followed up with some reflections about continuums, complexities and contexts, then some musing on patterns, power and systems, and most recently some thoughts about contexts, typologies and learning.
In early April, the plan (fingers crossed for synthetic systemic fortune!) is to see how these posts amount to more than the sum of their parts, and what insights the process of producing them has generated. But before then, we have question 4 to explore.
So, let’s go!
How might we further develop and encourage the use of approaches to evaluating and understanding change that are suitable for the complexities of living and learning systems?
Engagement: My re-sharing and amplification of Tom Aston’s excellent piece about the persistence of the damaging myth that experimental methods such as randomised controlled trials are a “gold standard” for evaluation, including for complex systems.
In too many cases, the approaches taken to understand the dynamics of change and to evaluate the impact of engagement in complex systems are - with their simplistic assumptions about causality, and their belief in the ability of experimental approaches to meaningfully isolate pathways to impact - the wrong tool for the job. In too many spaces and institutions, as Tom puts it, “the RCT-industrial complex has truly gotten out of control”.
The question is how to move beyond a methodological conflict that stubbornly persists, despite some encouraging signs five years ago with the apparent mellowing of Innovations for Poverty Action and J-PAL (see for instance, the generalizability puzzle, by Mary Ann Bates and Rachel Glennerster).
There are some grounds for hope. For instance: in the increased attention to causal pathways and non-experimental ways of trying to understand causality shown by some funders; in the increased respect for and attention to the value of emergent learning; and in the innovative approaches to evaluation being discussed in UNDP’s M&E Sandbox series (see for instance, the recent discussion of the experience of Luminate, the BHP Foundation and Humanity United).
Continued sharing of different organisations’ experience - the challenges they’ve faced as well as the progress they’ve been able to make - and whether it goes far enough (check the question/response from Toby Lowe and Kecia Bertermann at 1 hour, 21 minutes, 10 seconds of the recent sandbox video) will help. As will realism and pragmatism about the accountability constraints, reporting requirements and associated power dynamics, that many actors working within this space need to navigate; ignoring or dismissing questions about results and what works is not a good strategy.
Beyond this, careful and constructive systems leadership by a powerful player or two - step forward for instance, USAID, the Center for Global Development, the Bill & Melinda Gates Foundation and leading European donors (perhaps the UK and FCDO are still European and leading?), as well as innovative funders more embedded in the global south (perhaps UNDP’s M&E Sandbox can help to identify these?) - could make a big difference.
On which note, a recent post by Ruth Levine may well be of interest. Ruth is a long-time and powerful champion of evidence-based policy and the “what works” agenda, with a career that runs from multilateral development banks, the Center for Global Development, USAID, the Hewlett Foundation and most recently IDinsight, with a move to the Packard Foundation on the near-horizon (see Ruth’s wonderful explanation of her decision). In her always interesting Friday notes from 3rd March, Ruth reflects on the mis-placed envy amongst some in the global development space for what she refers to as the medicine model of how to get to policies that are more effective because they are evidence-based.
The alternative model that she outlines - a market of producers and consumers of policy advice - raises lots of questions for me, and/but also opens up the space for a much-needed discussion about how we might move beyond a failing medicine model for evidence-informed policy. As Ruth notes, the currently dominant evidence-based medicine model “is not helping us move the ball forward as fast as demanded by the crucial social and environmental challenges we face.”
The question of how to understand and evaluate complexity, and the question of what might be a better model for evidence-informed policy are not the same question. But they are very close cousins, with worldviews, evidence, learning and policy engagement as their grandfathers and grandmothers. I look forward to continuing to explore how those of us who share Ruth’s view, can continue to learn about the potential of different approaches to understanding and evaluating the complexities of living and learning systems and using evidence to improve them, and move the ball forward with the urgency required.
Next time, fingers crossed, systemic and synthetic reflections on awareness, emergence and learning!
As a side note / historical artefact, my 2011 take on the early enthusiasm for RCTs in revitalising the fight against poverty - a take that surprises me with its balance and is no doubt shaped by the ONE Campaign context I was in - can be found *here*. My earlier still reflections on complexity, emergence and measurement from a short-lived DFID perch can be found here.
To see my collated musings on awareness, resonance, engagement, reflection go here. To be kept in the loop for things I share in future, please sign up here.
First published on LinkedIn, 28 March 2023.