Mindset XR Module 6: How to clinically evaluate medical devices

Welcome to the Mindset Extended Reality (XR) for digital mental health programme learning resources, which include three series: medical regulation, clinical evidence and lived experience involvement. Mindset-XR is helping to catalyse the growth of immersive digital mental health solutions in the UK, through funding, tailored support and training. It is delivered by Innovate UK and the Health Innovation Network South London (HIN).

This series focuses on medical regulation, with key insights from Hardian Health. Across 10 modules, we provide an accessible introduction to people and companies that want to learn more about medical device regulation, with a focus on XR devices. Each module offers a high level overview of a different topic, including medical device regulation in the UK and EU, core medical device standards and overseas regulation. Each module includes additional resources to support your learning and a quiz to test your understanding.

Outline

Welcome to Module 6: Clinical Evaluation of XR Devices. In this section, we’re exploring the 5 key steps of clinical evaluation, in relation to XR devices. Topics include:

What is the clinical evaluation process?

Overview of the 5 key steps of clinical evaluation, in relation to XR devices.

What are the 5 steps of clinical evaluation?

A closer look at the 5 key steps of the clinical evaluation process, including definitions of each step.

Quiz

Multiple choice questions to test your understanding of the clinical evaluation process, in relation to XR devices.

What is the clinical evaluation process?

The Clinical Evaluation Process follows 5 key steps. These are:

Define Value
Define unmet need, value proposition

Conceptualisation
Define intended use (target market, conditions, users)
Scientific validity
Literature review, proof-of-concept studies
Analytical validity
Bench testing, usability testing, internal validation
Clinical validity
External validation, clinical investigations
Post Market
Post market surveillance, post market clinical follow-up

Deliver value
Demonstrate economic value

Before jumping into Step 1, it’s important to define the value you want to impart before you start the clinical evaluation process. This “Step 0” is your why. Eventually, you need to show that you can deliver that value (Step 6), and the 5 steps of clinical evaluation will help you get there.

What are the 5 steps of clinical evaluation?

In regulatory terms, conceptualisation is done by writing an Intended Use Statement. This is where you clearly define:

Intended medical indication e.g. mild or moderate depression.

Intended patient population e.g. english-speaking adults.

Intended user groups e.g. patients, clinicians, technical staff.

Intended parts of the body e.g. headset in contact with the face.

Intended use environment e.g. home or clinic setting, with a stable internet connection.

Operating principle:

Clinical: how does your product fit in the care pathway? At a more granular level, how does a user navigate through your product and use it?

Technical: how do you implement your product technically? What are the high-level architectures of any AI models you use? What is the process for any data transfer?

After defining your Intended Use, you need to also articulate the clinical benefit you aim to provide. This needs to be meaningful, measurable, and patient-relevant in terms of the outcomes related to a diagnosis, or a positive impact on patient management.

After defining your intended use, you need to demonstrate scientific validity. This is asking the question: is there a valid association between the product’s outputs and a clinical/biological target? For example, you may need to demonstrate the scientific validity of using VR-delivered therapies for the improvement of a particular mental health condition.

Evidence from scientific validity can come from two places:

Conducting a systematic literature review.

Any proof-of-concept studies you might have done. This is particularly important if the existing literature doesn’t support scientific validity.

After demonstrating scientific validity, it’s time to go and build the candidate software or product you want to put on the market. Analytica validity is the step of technically showing that your candidate product works as intended. This means:

Making sure all hardware components work.

Doing unit tests to show that each software requirement is met.

Showing that integration of hardware and software is adequate.

Conducting formative usability testing with users to make sure they are happy with the way user interfaces work, or interactions with the software are smooth.

If you are building your own hardware, this would also need to go through standardised requirements for:

Electromagnetic compatibility testing.

Radiofrequency testing.

Battery safety testing.

Biocompatibility testing of any surfaces and materials that come in contact with the body.

Once you have proven analytical validity (the technical capability of your product), it’s time to prove clinical validity (the clinical capability of your product).

Often, you’ll have to conduct a clinical investigation to do this. A clinical investigation is a study that should have a robust design that aims to prove the benefits you defined in step 1. This has to be done on an independent set of participants. If using AI, all validation also needs to be done on independent data.

You may only need one clinical investigation for regulatory approval, as long as it’s designed well, and it proves the claims you make for your technology.

At this point, you’d also conduct more formal, summative usability testing, where you show that the human factors that might affect the use of the device don’t affect its safety or performance significantly. This is usually done with 15-20 representative users of the software.

Investigation design

The design of your study design depends on the claims you want to make. It’s important to define whether you want to claim to be:

Superior to the standard of care.

Non-inferior to the standard of care.

An adjunct to another intervention.

For interventional studies, it’s likely you’ll need a prospective study with an appropriate control arm (e.g. waitlist or active control). For diagnostic studies, it’s sometimes possible to use retrospective data, but in either case the reference standard you are comparing against needs to be well-established.

If your clinical investigation is successful, this would be the point where you would submit your documentation for regulatory approval, or self-certify that you are compliant with the regulations. But clinical evaluation must continue after regulatory approval too. These post-market activities can be split into:

Post-market surveillance:

Collecting user feedback (actively and passively).

Conducting surveillance for safety and adverse effects.

Monitoring and addressing complaints e.g. bugs, interface issues.

Post-market surveillance:

Check that the performance of your devices is maintained in the real world. This might mean conducting studies post-market, or continuously monitoring a few users every few months to ensure adequate performance.

Monitor for performance drift of any AI-driven components.

Assess for variability between populations and deployments, and address any findings that arise.

Summary

In this module, Clinical Evaluation of XR devices, we explored clinical evaluation, in relation to XR devices. After using this resource, you should have a understanding of the key 5 steps of clinical evaluation:

Define your intended use.

Demonstrate scientific validity.

Demonstrate analytical validity.

Demonstrate clinical validity.

Continue demonstrating safety and performance post-market.Â