Impact evaluation should have a central role in evidence-based policymaking. But, whilst the USAID Health Bureau has prided itself on taking international leadership in evidence-based results monitoring and comprehensive evaluations, attention to evaluation has declined in recent years.
The earlier technical excellence of the USAID Health Bureau in evaluation methods was demonstrated by the development of the LOG FRAME in the late 1970s and subsequent development documents (TIPS: http://evalweb.usaid.gov/resources/tipsseries.cfm) and operational guidance for Agency operations (Center for Development, Information and Evaluation (CDIE), USAID, www.USAID.gov/us; Automated Directives System (ADS), USAID, www.USAID.gov/policy). But since 1994 there has been a steep decline in the quantity and quality of evaluation; the number of evaluation studies falling from a peak of 497 a year to a low of 104 in 1998 (Clapp-Wincek and Blue 2001). This decline has been attributed to many factors, including Albert Gore’s policy change on ‘reinventing government’, a sharp reduction in technical staff, and a change in USAID Guidance from requiring every project to be evaluated to recommending that evaluations only be done in response to management need.
Since 2000, there have been several Agency-wide and Bureau-specific reviews of evaluation (e.g. ClappWincek and Blue 2001; and Weber 2004), which have demonstrated the loss of institutional learning and best practices. While the country USAID missions depended on evaluations, their greatest concern was the very limited number of in-depth programme evaluations. Moreover, while the partners did many of the evaluations, USAID did not (Clapp-Wincek and Blue 2001: iii), and those few evaluations supported by the missions were not being submitted to the Development Exchange Clearinghouse (DEC).
Evaluation quality has long been an issue with USAID, both Agency-wide (e.g., Hopstock et al. 1989) and in the technical bureaux (e.g. Adamchak et al. 2004 for Health; Bollen et al. 2005 for Democracy and Governance). Many evaluation reports do not include more than a few paragraphs on method, and many were qualitative and unsystematic: the expatriate ‘fly-in’ assessment where a team comes for 2–3 weeks and bases its findings only on qualitative interviews with key informants and stakeholders. Most reviews of USAID evaluations found weak methods employed, even with external, professional evaluators whose objectivity was often compromised by their desire to please the managers and continue to be hired. This article documents these trends in recent years, analysing the factors behind them and the steps required to ensure the production of more and better impact studies.
The focus of this study is on the methodological strength and design rigour of evaluations of outcomes, effects, and impacts. White (2006) defines impact as the counterfactual analysis of how an intervention affects final welfare outcomes. In that sense, we want to know if the donor-funded activities are attaining their expected results as set out in the project paper and the results framework or the monitoring and evaluation (M&E) design. However, since donors fund too few real impact evaluations of project attribution and the counterfactual, evaluators have often been limited to considering whether the project achieved its intended outcome in its intervention areas or groups, preferably as compared to control groups. Moreover, in order to inform future programming, evaluation must be transparent and externally credible to decision-makers. However, without the rigour of an impact evaluation these attributes are harder to achieve.
Related Content
This article comes from the IDS Bulletin 39.1 (2008) Lost Opportunities and Constraints in Producing Rigorous Evaluations of USAID Health Projects, 2004‐7