Mindset XR Module 6: How to clinically evaluate medical devices

In regulatory terms, conceptualisation is done by writing an Intended Use Statement. This is where you clearly define: 

• ---

  Intended medical indication e.g. mild or moderate depression.

• Intended patient population e.g. english-speaking adults.

• Intended user groups e.g. patients, clinicians, technical staff.

• Intended parts of the body e.g. headset in contact with the face.

• Intended use environment e.g. home or clinic setting, with a stable internet connection.

• Operating principle:

  • Clinical: how does your product fit in the care pathway? At a more granular level, how does a user navigate through your product and use it? 
  • Technical: how do you implement your product technically? What are the high-level architectures of any AI models you use? What is the process for any data transfer? 

  ---

After defining your Intended Use, you need to also articulate the clinical benefit you aim to provide. This needs to be meaningful, measurable, and patient-relevant in terms of the outcomes related to a diagnosis, or a positive impact on patient management. 

After defining your intended use, you need to demonstrate scientific validity. This is asking the question: is there a valid association between the product’s outputs and a clinical/biological target? For example, you may need to demonstrate the scientific validity of using VR-delivered therapies for the improvement of a particular mental health condition.

Evidence from scientific validity can come from two places: 

1. Conducting a systematic literature review. 

2. Any proof-of-concept studies you might have done. This is particularly important if the existing literature doesn’t support scientific validity. 

After demonstrating scientific validity, it’s time to go and build the candidate software or product you want to put on the market. Analytica validity is the step of technically showing that your candidate product works as intended. This means: 

• Making sure all hardware components work.

• Doing unit tests to show that each software requirement is met.

• Showing that integration of hardware and software is adequate.

• Conducting formative usability testing with users to make sure they are happy with the way user interfaces work, or interactions with the software are smooth.

  ---

  If you are building your own hardware, this would also need to go through standardised requirements for: 

• Electromagnetic compatibility testing.

• Radiofrequency testing.

• Battery safety testing.

• Biocompatibility testing of any surfaces and materials that come in contact with the body.

Once you have proven analytical validity (the technical capability of your product), it’s time to prove clinical validity (the clinical capability of your product).  

Often, you’ll have to conduct a clinical investigation to do this. A clinical investigation is a study that should have a robust design that aims to prove the benefits you defined in step 1. This has to be done on an independent set of participants. If using AI, all validation also needs to be done on independent data.  

You may only need one clinical investigation for regulatory approval, as long as it’s designed well, and it proves the claims you make for your technology. 

At this point, you’d also conduct more formal, summative usability testing, where you show that the human factors that might affect the use of the device don’t affect its safety or performance significantly. This is usually done with 15-20 representative users of the software. 

Investigation design 

The design of your study design depends on the claims you want to make. It’s important to define whether you want to claim to be: 

• Superior to the standard of care.

• Non-inferior to the standard of care.

• An adjunct to another intervention.

  ---

For interventional studies, it’s likely you’ll need a prospective study with an appropriate control arm (e.g. waitlist or active control). For diagnostic studies, it’s sometimes possible to use retrospective data, but in either case the reference standard you are comparing against needs to be well-established. 

If your clinical investigation is successful, this would be the point where you would submit your documentation for regulatory approval, or self-certify that you are compliant with the regulations. But clinical evaluation must continue after regulatory approval too. These post-market activities can be split into: 

Post-market surveillance: 

• Collecting user feedback (actively and passively).

• Conducting surveillance for safety and adverse effects.

• Monitoring and addressing complaints e.g. bugs, interface issues.

Post-market surveillance: 

• Check that the performance of your devices is maintained in the real world. This might mean conducting studies post-market, or continuously monitoring a few users every few months to ensure adequate performance.

• Monitor for performance drift of any AI-driven components.

• Assess for variability between populations and deployments, and address any findings that arise.