Rapid evaluation in the NHS: learning to move quickly without rushing

The NHS loves a bit of innovation. It is prolific at launching new services, redesigning pathways and introducing new digital tools.

Yet when it comes to understanding the impact of these changes, formal evaluations are often hit and miss, if they happen at all. This is in part because evaluations take time, money and specific evaluation expertise, something the NHS often lacks.

Even when evaluations are commissioned, they are either done way too early so they end up showing little impact, or they take way too long meaning that successes are hard to replicate and unintended consequences are left to grow unchecked. Sometimes policy thinking just simply moves on, and the evaluation’s findings never get a chance to shape implementation.

But it doesn’t have to be this way. During my time at National Voices I have had the great pleasure of working with BRACE (Birmingham University, RAND Europe and Cambridge Rapid Evaluation Centre) which is one five National Institute for Health and Care Research (NIHR) commissioned rapid evaluation centres which are breaking the mould.

Their work shows that rapid evaluation is not simply “normal evaluation, but faster”. It is a different way of thinking about evidence: more iterative, more responsive and more closely connected to the realities of implementation.

The work of the centre follows the four Rs: rapidity, responsiveness, relevance and rigour. That final R perhaps matters most. After all, speed without rigour is not evaluation; it is just enthusiasm.

Looking at our work so far with BRACE, I have three important reflections.

1. Healthcare needs more iteration

There are a lot of pressures on the NHS to adopt new technologies. There is far less pressure to adopt some of the ways of working tech companies have pioneered.

For example, tech companies rarely imagine that the first version of a new innovation will be perfect. They launch a minimum viable product, observe how people use it, identify friction, fix what is not working and release a better version. In contrast I have found the NHS tends to be less “design, learn, adapt”, and more “design, launch, defend”.

The healthcare sector’s hesitance is to work in this way is understandable. The consequences of getting things wrong can be serious, and therefore constant iteration feels inherently dangerous. But neither should we fall into the trap of thinking the status quo is always safe. Sometimes it is simply that problems go unnoticed for longer.

As healthcare becomes increasingly shaped by technology – artificial intelligence, virtual wards, digital triage and data-driven decision-making – the NHS will need evaluation systems that can keep pace. It is no longer enough to ask, years later, whether an intervention “worked”. We also need to ask:

• Is it being implemented as intended?
• Who is using it, and who is not?
• What unexpected burdens is it creating?
• Is it safe?
• What needs to change before it is scaled?

The BRACE and RSET[1] teams’ work during the pandemic to evaluate remote home monitoring using pulse oximetry is a good example of what rapid evaluation can achieve. An initial phase was completed in two months to inform national rollout, followed by a longer and more detailed evaluation of effectiveness, costs, implementation, inequalities and patient experience.

Rapid evaluation can provide feedback loops while an innovation is still malleable. It doesn’t just help us work out if something is working or not, the very act of rapid evaluation can help the NHS improve the programme it is evaluating in near real-time.

2. Patients are the earliest warning system

The second reflection is perhaps the most important. Rapid evaluation works best when patients are involved, not as decorative participants added to the final page of a report, but as people helping to shape the questions and approach from the beginning.

Patients often detect the difference between a service that looks good on paper and one that works in real life. They notice when a digital pathway assumes reliable broadband, when a remote monitoring device creates anxiety, when a telephone system excludes people with hearing difficulties or when a supposedly convenient service simply moves work from the NHS to the patient and their family.

These are often the earliest signs of success or failure.

BRACE projects have consistently demonstrated how patient and public contributors have helped shape research questions, review recruitment materials, interpret emerging findings and decide how results should be communicated. Even more important, the work hasn’t taken an “any patient will do” mentality. Take the current evaluation of the NHS Perinatal Equity and Anti-Discrimination Programme. The BRACE team has recruited National Voices member, Cysters, to jointly lead the PPIE for the study, ensuring people from the impacted communities are put at the heart of the evaluation process.

This work isn’t always easy, and meaningful involvement takes time, which can run at odds with the idea of rapid evaluation. But this is where the partnership with National Voices has been so critical, helping BRACE colleagues find the right VCSE organisations to work with. Crucially, National Voices then acts as trusted broker, facilitating relationship building so that evaluation teams can work quickly and meaningfully with people with Lived Experience.

3. Evaluate at the right point in the cycle            

There is a danger in becoming too excited about speed of adoption of new technologies and ways of doing things. The NHS can sometimes behave as though a new intervention, launched on a Wednesday morning, has failed if it has not transformed the world by Thursday afternoon.

Innovation is much more of an adoption curve than a light switch.

A service may initially be slow because staff are learning new roles, patients are unfamiliar with it, data systems are incomplete or the surrounding administrative and service infrastructure has not caught up. Early evidence may reveal important implementation problems without (or before) telling us whether the underlying idea has long-term value.

The BRACE work makes this point clearly, rapid evaluation does not necessarily mean evaluating everything immediately. It may mean evaluating early implementation, mobilising research quickly, sharing findings in stages or embedding short feedback cycles within a longer study. Ultimately this can all feed into longer term implementation, adaptation and learning.

The crucial question is not simply “How quickly can we evaluate this?”. We should be asking “when will the evidence mean something?”.

That requires proper scoping of each evaluation from the outset. Before collecting data, policy makers and evaluators need to understand what the intervention is really meant to achieve and what outcomes could plausibly appear within the timeframe.

Sometimes the correct rapid conclusion is that it is too soon to assess impact. In the case of the BRACE Centre’s evaluation of women’s health hubs, for example, early work found considerable variation in what a “hub” actually meant. Rather than pretending that a definitive impact assessment was possible, the evaluation focused on mapping the different models, understanding implementation and preparing the ground for longer-term study.

This is not a failure of rapid evaluation. It is one of its most valuable outputs … preventing premature certainty.

Growing the approach

In my view, a more wholesale adoption of rapid evaluation principles could help create a health service that adopts innovations earlier, listens more carefully, learns continuously and is honest about uncertainty.

I am therefore encouraged by the growing level of interest in the concept amongst national policy makers, and as I leave National Voices to take up my new role I will certainly be a champion.

The key now is to make sure that whatever the innovation – be it virtual wards, advice and guidance or ambient voice technology – that rapid evaluation becomes a mainstream concept. This would put patients and communities truly at the heart of the innovation life-cycle.

[1] The Rapid Service Evaluation Team based at UCL and the Nuffield Trust – RSET: rapid service evaluation team | Nuffield Trust