How might evaluation research respond to the complex and emergent nature of holistic community-led development? What does an equitable living partnership between evaluators and researchers, funders and programme implementers look and feel like? What are the highs and lows of navigating complexity, power and emergence in evaluation practice?
These are some of the questions we grapple with in this reflective blog series. Too often, the decision making behind evaluation design and practice remains hidden from view, feeding a fallacy that evaluation is a neat and tidy endeavour that is best left to the ‘experts’.
In this first blog, we introduce the work of the Full Spectrum Coalition evidence and learning group and the challenge it responds to. In our next blog we reflect on early conversations about what evaluations of holistic community-led programmes could look like, and how deliberation can support grappling with uncomfortable decisions, where axiological and epistemological stances explicitly or implicitly drive our biases.
Evaluation is a pragmatic science that is, above all, use- and action-oriented. We as evaluators and researchers are drawn to the promise that critical evaluative reasoning can support robust examination of development programming to inform more grounded, equity-oriented and evidence-based assumptions about how change happens.
This promise, however, can seem like a mirage at times. A vision you can almost make out, a place where programme implementers and evaluators with a range of expertise agree on the objectives and questions and deliberate on the most appropriate, credible, and ethical designs. These three criteria (explained in Box 1) set the scene for the quest for a ‘perfect’ design.
| Box 1. Criteria for evaluation design |
|
Credibility The design should produce credible findings. We adopt the common definition of credibility found on Better Evaluation referring “to the trustworthiness of the evaluation findings, achieved through high-quality evaluation processes, especially rigour, integrity, competence, inclusion of diverse perspectives, and stakeholder engagement.” Appropriateness By ‘appropriate’, we refer to designs that respond to the evaluation questions, are using available methodologies that are suited to the questions and to the specific programmatic attributes (based on the design triangle introduced by Stern et al. in the DFID Working Paper 38). This requires balancing real-life constraints of programme implementation, capacity and resourcing with the desire to build evidence and theory. Ethics By ethical, we refer to study designs that not only respect international ethical rules of engagement (such as the Helsinki declaration), but that deeply wrestle with moral principles of conducting research and agree on a justification of their design choices in ways that align with respecting rights, dignity, and well-being of all participants. |
One reason it remains a mirage rather than reality is because evaluation research is not usually designed through a deliberative process. Indeed, the primacy of independence can have the opposite effect. Questions are predefined by the ‘users’ (generally understood as those commissioning an evaluation) – usually in the form of a long aspirational shopping list (kitchen sinks often appear) rather than a prioritised list to support a strategic learning agenda.
In response, a methodological design is proposed by the ‘producers’ of evidence – usually driven by a combination of our conscious and unconscious axiological and epistemological biases (hammers and nails come to mind here). While some level of interaction and negotiation might ensue, distinct roles are most often adhered to, leading to a performative dance. It rarely feels like a process that respects the dignity of all the people in the room. And those impacted by the programmes being evaluated, and whose experiences of change are central to the evaluation, are usually not even in the room.
When designing evaluation research for programmes that support participatory and community-led processes, there are multiple dynamics that call for disrupting the performative dance to get us beyond the mirage:
This is the daily reality of three organisations – One Village Partners, Legado and Tostan – who formed the Full Spectrum Coalition, together with Pastoral Women’s Council, Planet Indonesia and Health in Harmony. What unites them is a Holistic Community-led Development (HCLD) approach to creating lasting change, driven by social transformation rather than a narrower focus on sector specific human development impacts (in health, education etc.). They came together in the coalition to explore how their programmes work through a multi-year and multi-country evaluation research initiative with the aim of creating new partnership opportunities, generating innovative scientific understanding of how HCLD works, and informing better decision-making by funders, practitioners and policymakers. IDinsight and IDS (through the Centre for Development Impact) will serve as research partners, bringing a range of methodological and thematic expertise in designing, conducting and supporting the use of evaluation research.
The five partners have been collaborating to fundraise for the joint initiative and to begin designing our evaluation and learning agenda. You can read more about our vision and intended outcomes.
In our next blog, read our reflections from early deliberations on evaluation design – seeking to achieve credibility, appropriateness and ethics.
14 February 2024
