Post Title

Building a reliable evidence base is an important consideration for any innovator in the healthcare space. In this blog, HIN Head of Insights Dr Andrew Walker provides a high-level introduction to evaluation and evidence-generation in the context of the Innovate UK Mindset-XR Innovation Support Programme.

One of the most exciting things about the Mindset-XR programme is how we are bringing together experts from so many different fields to make a difference to mental health; This includes supporting innovators with backgrounds in technology, the creative arts, education, media and gaming.

While this diversity of thought and perspective is a strength when it comes to finding creative solutions, it also poses some challenges for people less familiar with healthcare as a sector. One of the biggest challenges – and something which we’ve already heard investors flag – is around innovators understanding the importance of robust and compelling evidence for the safety and effectiveness of their XR-based intervention.

Without the right types of high-quality evidence, scale and spread of any commercial innovation is nigh-on impossible. This is especially true in the current economic environment. Commissioners, payers and patients need to be highly confident they understand how a technology will perform before they even consider paying for it.

As a result, the team that I head up at the HIN – covering evaluation, informatics and health economics – is extremely busy. We’re also involved in delivering DigitalHealth.London’s Evidence Generator Bootcamp, which has seen an explosion in interest recently.

Evaluation and evidence generation is a complex field, and the HIN south London Mindset XR Innovator Support Programme  will be providing bespoke support for innovators taking part, along with in-depth online learning resources. The purpose of this blog is to  provide a very brief overview of what the evidence generation landscape looks like in the UK, and start innovators down the right path when it comes to collecting the evidence they need for their work to succeed.

What do we mean by evidence and evaluation?

Although specialists working in the field have strict definitions for what these terms mean, you might hear them used interchangeably. Broadly speaking, it can be helpful to think of them as the below:

• Evaluation: judging the worth or value of something, often in comparison to other available options. For example, you might seek to evaluate whether your innovation delivers better clinical outcomes than the current treatment approach.
• Evidence: the sources of information that you might use to make a decision about something. This could include evaluations, feedback from patients or users, analytics or analyses of population health.

Within health and care innovation, it is generally considered best practice to integrate evaluation and evidence-related activities into every stage of your projects. Your stakeholders will want to see that you have used evidence to understand a particular disease area to specialise your solution for; there will be an expectation that you have methods for measuring how your innovation works with users (such as analytics within your XR software); to drive widespread adoption will eventually need to have compelling evidence that your innovation provides advantages over what is out there already.

As you start to look at what evidence and evaluation might mean for your innovation, the complexity and costs associated might begin to look quite daunting quite quickly – especially for more complex formal evaluations with patients such as clinical trials.

Luckily, in recent years, the path for developing good-quality evidence for digital health interventions has become much clearer.

The role of NICE and the MHRA

There are two key organisations who immersive technology innovators need to be aware of when it comes to defining national standards for evidence:

• The National Institute for Health and Care Excellence (NICE)
• The Medicines and Healthcare Regulatory Agency (MHRA)

Probably the best starting point for most digital innovators is to look at NICE’s Evidence Standards Framework (ESF). The ESF is intended to be a shared framework which allows innovators and commissioners to understand the most important types of evidence for different digital health technologies (DHTs).

Using the ESF, DHTs are grouped into three tiers based on the intended use of the innovation. While the range of applications for XR in mental health is almost limitless, we anticipate most innovations we come across during the Mindset programme will fit into Tier C (DHTs for treating or diagnosing medical conditions) or Tier B (DHTs which help people to manage their own health and wellness).

After the correct tier has been identified, the ESF lays out 21 standards for consideration, ranging from clinical effectiveness to environmental sustainability.

Because the ESF is a framework rather than a formal assessment, it does not prescribe exact evidence requirements for each of these standards. Instead, it provides a comprehensive overview of all the different evidence areas which are likely to need to be considered. Within these, companies can develop their evidence base over time as their innovation scales.

The MHRA’s role is different – they regulate (by law) all medical devices in Great Britain and Northern Ireland.

Defining what is and is not a medical device in the world of DHTs is not always straightforward; the MHRA have produced guidance on digital apps which may help to establish whether your technology falls under their remit. You can also contact them for additional guidance.

If your immersive technology is classed as a medical device, you will need to meet strict standards around safety and efficacy, including carrying out a clinical investigation. Meeting these standards is a significant undertaking and you should begin planning to meet MHRA certification criteria as early as possible in the development of your product.

Stay informed

Although evaluation can seem like a highly complex area of innovation to navigate, there is support available.

The Mindset-XR Innovator Support Programme provides specialist support – including advice on evaluation – to companies on our programmes. The team will also be producing publicly available learning materials about regulation and evidence generation for immersive technologies in the coming months; please do sign up to our newsletter if you want to be the first to get access to them.

Elsewhere, regional health innovation networks also provide support for innovators and can give you impartial advice on evaluation, as well as helping with connections to evaluation specialists.

Another useful resource for getting practical information on how to approach evaluation/evidence generation is a collection of resources on evaluating digital health products produced by the Office for Health Improvement and Disparities.