What should safe AI in healthcare look like?

On 10 September, the National Commission into the Regulation of AI in Healthcare published its recommendations.  

National Voices supported the Commission by delivering engagement with young people, unpaid carers and people with a learning disability, exploring what would give them confidence in AI-supported healthcare. 

The Commission generally found conditional support for AI. That broadly matches what we heard: acceptance depends on the technology’s purpose, the consequences of error, whether mistakes can be corrected and who is responsible. 

Safe, fast and trusted must work together 

The use of AI in health could support earlier diagnosis, improve administration and give professionals more time with patients. But efficiency gains are only valuable when AI implementation is safe, useful, inclusive and reliable in real-world scenarios. Accelerating adoption without the necessary infrastructure, workforce and patient buy-in, or safeguards risks scaling existing inequalities alongside innovation. 

Different uses of AI also create different risks. Applications deemed (or perceived!) ‘lower risk’ must not mean that no protection is required. Documentation errors, omissions and hallucinations can enter clinical records and shape later decisions, while accuracy of tools like AI scribes can vary across accents, languages and communication needs. 

Proportionate regulation should therefore establish a universal baseline of transparency, consent, accuracy monitoring and clinician sign-off, with stronger evidence and oversight where errors could have serious consequences. 

Trust is earned through experience 

Public trust cannot be created by describing a system as trustworthy. It is built through people’s experiences of care. 

Communities we spoke to asked for accessible information about when AI is used, what it does and what choices they have as a patient or carer. They wanted human oversight and clear routes to challenge decisions, report concerns and seek redress. Their questions were practical: who is responsible, what information is used, whether a professional can override the result and what happens when technology fails. 

The Commission recommends greater transparency, opt-out where “possible or appropriate”, routes to redress, a public incident database and ongoing public involvement. I welcome these commitments and am especially encouraged to see recommendations around taking a coproduction approach with patients and the public when it comes to developing user-centred process and products. Patients should help define when opting out is genuinely not possible or appropriate; and lines of accountability must never be unclear for patients. 

Equity is part of safety 

AI may perform differently across skin tones, accents, languages and communication needs. Digital-only routes may also exclude people without suitable devices, connectivity, confidence or support. These are safety and access issues. 

The Commission treats health equity as part of safety and performance but proposes guidance for manufacturers. I believe that equity should instead be a condition of approval and continued use of AI-enabled healthcare products. Developers and healthcare organisations should test technologies with people most likely to be excluded and/or experience health inequalities, monitor outcomes across communities and act when inequalities emerge. This process and resulting actions taken to address inequalities should ideally be made public so that affected communities can be reassured. 

Oversight must continue 

AI can change through updates, new data and use in different settings. A decision made before deployment cannot provide lasting assurance. Approval must be followed by real-world monitoring, clear escalation processes and swift action when evidence of harm emerges. Patients and communities should remain involved to identify problems performance measures may miss. 

Human oversight and access to a person/clinician when requested were conditions of acceptance in our engagement, particularly for decisions about diagnosis, medicines or urgent care. Government and the MHRA should define where oversight is required, address automation bias and protect non-digital routes to care. 

The test is simple:  

• Can people understand when AI is involved and what choices they have? 

• Can people get to a human quickly when it matters? 

• Is accountability and redress clear when harm happens? 

• Will this reduce inequalities rather than widen them? 

• Is there continuous monitoring in real-world use, with patient feedback built in?⁠⁠ 

Read the Commission’s recommendations: https://www.gov.uk/government/publications/national-commission-into-the-regulation-of-ai-in-healthcare-recommendations-for-a-future-regulatory-framework  

Read National Voices’ engagement report: https://www.nationalvoices.org.uk/publication/mhra-ai-and-regulation-engagement-report/