In this blog written from a meeting of thousands of knowledge brokers and scientists at the Global Evidence Summit, I ask how are we going to put the rapid into rapid evidence reviews?
I was excited to attend a session at the Global Evidence Summit in Prague this week on “Meeting the Need for Speed”, organised by global health evaluators Cochrane. The workshop gave participants the opportunity to learn about new recommendations on conducting rapid evidence reviews. However, it raised some tricky questions around the trade-offs between speed and rigour.
What do we really mean by rapid?
The new guidelines conform closely with Cochrane’s rigorous systematic review guidance. Participants were asked to assess the strengths and weaknesses of one such review and my break-out group of rapid reviewers were surprised to find it had incorporated seven databases, with six reviewers working independently to perform screening and two others reviewing all citations. Overall, it was an impressively rigorous review completed in six months. The compromise on rigour seemed marginal thanks to a deeply experienced and impressively resourced team.
However, there were two snags with this for me. Firstly, the secret sauce for this accelerated process (a systematic review can take almost two years!) was the sheer capacity of the team. Six reviewers worked on this simultaneously. This is an amazing level of resource. Secondly, in my experience, when policy makers talk about needing evidence rapidly, they are thinking day or weeks and not months.
Approaches to hyper fast reviews
In the Knowledge for Development and Diplomacy (K4DD) programme, led by the Institute of Development Studies (IDS), our Rapid Evidence Reports are delivered to the UK Government in under a month. Sometimes they are required in days, such as in response to the current mpox crisis. Even when a bit more time is provided, so we can produce a more detailed report that critically assesses the quality of the evidence, the budget is not sufficient to mobilise a large team of methodology and topic experts simultaneously.
The demand for hyper rapid evidence summaries is higher than ever. Between 2016-2022 the former FCDO Knowledge for Development Programme (K4D) produced a staggering 1200 reviews for the UK Government. This included a special service entirely dedicated to Covid-19. Since the launch of the successor K4DD programme in December 2023, we are 150 reviews in and counting. These reviews are highly policy relevant and produced by a combination of a standing team of expert rapid reviewers and a global network of consultants with the necessary subject matter expertise. These products have their limitations and our clients fully understand this. We often get questions which prompt us to refer them to other more appropriate services such as rapid evidence assessments that take around six months or even systematic reviews.
Innovations in rapid reviews for policy
There is growing interest in an in-between service that is faster (and cheaper) than a six-month process but more rigorous than our super-fast service. The challenge will be the resourcing and the management of expectations. You can get an excellent product in three months that pays attention to at least some of the Cochrane guidelines around restricted methods, risk of bias assessment and certainty of evidence. However, it means throwing some significant researcher time at each review and a willingness for some trade-offs between academic rigour and policy relevance.
There are others doing interesting work in this space who are grappling with the same challenges. At the Global Evidence Summit we heard from Jitka Klugarová from the Institute of Health Information of the Czech Republic, who warned that we must learn how to manage the delicate balance between speed and rigour. Much of the innovation in rapid reviews for government is coming from the low and middle-income countries where approaches frequently value coproduction with policy communities. We heard today from Promise Nduku at the Pan African Collective for Evidence, who reminded us that rapid reviews need to be shaped by the evidence users. Likewise, Uganda’s Centre for Rapid Evidence Synthesis (ACRES) has pioneered new ways to rapidly connect evidence with policy that are built on authentic research policy partnerships.
When is the evidence good enough?
I am keen to explore new approaches, but it is not clear how relevant new guidelines for rapid reviews, such as Cochrane’s, are, given the fast-paced policy orientated and demand driven contexts some of us work in.
Artificial intelligence may offer one potential way forward and just about everybody at GES is talking about this. Nonetheless, in my experience the key to a good rapid service is the human-to-human interaction between the client and the research team. The requester needs to be willing and available to invest some time in helping to shape the review, be willing to compromise on breadth versus depth and ensure it is the right option for them.
In a world rocked by crisis, increasing epistemic uncertainty and global challenges, how do we decide when the evidence is good enough? A gulf remains between rigorous evidence synthesis and the kinds of rapid reviews increasingly needed by policymakers. Cochrane has done well to try and overcome its community’s anxiety around a watering down of world-class standards to try and meet this challenge. However, its proposed solution is far from being the kind of fast and cost-effective process that is needed in so many sectors and places.
Overcoming our differences to find a way forward
I hope that diverse communities of knowledge users, knowledge intermediaries and researchers engaged in demand driven evidence synthesis can work together to deliver the necessary innovations. It will mean overcoming our methodological differences and getting donors and partners to understand the opportunities and limitations of different approaches.
To find out more about the Knowledge for Development and Diplomacy rapid evidence programme go to www.k4d.ids.ac.uk
IDS also runs a Shaping Policy with Evidence short course, designed to help participants make better use of evidence to tackle global challenges.