Welcome to the Mindset Extended Reality (XR) Innovation Support Programme learning resources, which include three series delivered in conjunction with our expert Mindset-XR programme partners:
This series focuses on research and clinical evidence, with key insights from King's College London's Institute of Psychiatry, Psychology and Neuroscience. Across a number of modules, these resources will guide you through your research journey, from establishing what you plan to investigate, to conducting research and disseminating your findings.
Outline
Welcome to Module 17: Data collection. This module will be introducing the processes and considerations relevant to data collection in research.
In this section, we're focusing on:
Types of data
Understanding the types of data that can be collected for XR research.
Methodology
Identifying the common methodologies used in XR research, and considering what measures are most adequate and effective for your research.
Recruitment tips
Aspects of recruitment, and groups of people you have to consider when recruiting for your research study.
Types of data
The type of data to be collected in a research trial is linked to the aims of the trial. Data can answer questions on:
This refers to the measure and protocols implemented to protect the wellbeing of research participants, researchers, and the environment.
For participants and researchers, this is the extent to which research projects, methods, and outcomes are viewed as appropriate and suitable.
To other bodies like ethic committees, and funding bodies in the wider community, data gives an indication of if the study is feasible. It can show how practical, and how likely the research is to be successful within the constraints of context and time.
Data on user experience is particularly important within the context of XR research.
The data can help us understand and evaluate how users interact with the product, or the system, and it can help us to make optimisations to make their experience more user friendly.
Efficacy is one of the key reasons we collect data.
It can give us insight into how a treatment can produce the desired effect under ideal conditions, typically in randomised controlled trials or laboratory settings.
Slightly distinct from efficacy, is effectiveness. This refers to the ability to detect the effect of an intervention or treatment in real life settings, not under controlled conditions.
Data can also give us information on the cost of a treatment, and it's cost benefit.
Methodology
When deciding what method to use, it is key to consider what would suit the research needs best.
Among the different types of methods you can choose for your research (as seen in module 14) qualitative and quantitative methods are the most used in XR research.
There are merits to both approaches, and the choice will depend on research objectives.
This method involves interviews and research groups.
It is better suited when developing an application, or when needing to refine it.
For example: you can interview people that use a prototype of an XR/VR product you've developed, and using those insights make necessary changes to the product.
This method involves experiments and surveys. Quantitative information is often used by commissioners and panels informing clinical guidelines to make recommendations.
It is considered the gold standard for efficacy evaluation. These are the types of methods preferred when conducting randomised controlled trials. This is because these methods can quantify change in a numerical fashion, and this is considered better suited for comparing results between different participants.
For example, participants are allocated to different groups as part of the randomisation process, yielding different sets of results.
A research project can use both quantitative and qualitative methods.
It's not the case that you have to choose between one or another, projects that use both methods are called mixed method projects. They often gather insights from participants from interview sand direct feedback, as well as collecting data from numerical outcomes and measures.
Measures
There are different types of measures that can be used in XR research, including:
These tend to be collected automatically by the technology, and it can be like the start and in time of an experience, a bottom press, or eye tracking.
These are collected from the participant directly, and include: questioners, self-administered, but also interviews which require an interviewer collecting information.
These are performance measures such as aptitude tests, reaction time, response tests, or neuropsychological tests.
There are the metric and physiological data that can also be collected alongside the virtual reality experience and integrated in the data collection plan.
Cultural sensitivity
When selecting research assessment tools, we need to make sure these are culturally sensitive to ensure that these are appropriate and effective for diverse populations.
To help minimise cultural biases, it is important to ensure the content of assessment tools is relevant to the cultural context of participants. It must reflect norms and values, whilst minimising misunderstandings. This includes:
Language
Relevant examples
Scenario considerations
Minimise the use of acronyms
If a translation is needed, make sure it is an accurate one that involved people who are native speakers, and ensure back translation measures.
Consider if there is data on how the measure performs in different groups.
For instance, there are published validation studies in different settings. This might be useful to consider.
Conduct pilot testing with a sample from the target cultural group to identify potential cultural biases or issues, and use feedback to make the necessary adjustments if needed.
It is important to ensure that the measure and the language use is inclusive and represents the diversity within the cultural group, including considerations of:
Age
Gender
Socioeconomic status
Other relevant protected characteristics
It is imperative to train whoever is collecting the measure in cultural competence to ensure that they're aware of cultural differences and can interact respectfully and effectively with participants from a diverse background.
Engage cultural expert and communities to support research, training and measurements.
Adaptation is also something useful to do. Good measures tend to be adaptable to different cultural contexts. They should be able to do that without losing their core purpose or validity.
Lastly, regularly evaluate and update the measure based on your research feedback and changes in the cultural context.
Primary and secondary outcomes
A research project can have different types of outcomes - this is normal and welcome. However, it is good practice to try to define a primary outcome.
This is the main outcome the study is measuring, and in the case of an intervention study, there tends to be one primary target outcome of the intervention.
In certain cases is possible to have more than one primary outcome.
The reliability on how we can detect change on an outcome is often a factor by the number of participants that is included in a study. Generally, the more participants, the more precise the effect of an intervention. Therefore, it is good practice to consider carefully participant number and effect sizes detection ahead of starting an efficacy trial. The input of a statistician or expert in trials might be needed.
It is possible to have multiple secondary outcomes. Ideally they should cover different constructs - meaning that it is not measuring the same thing, and in particular, not overlapping with the primary outcome.
How to choose a good measure?
It is important to choose measures with good psychometric properties - it is a word that means that they are sound, from a statistical point of view, reliable and accurate.
The key psychometric parameters are:
Accuracy
is the degree to which a measure correctly reflects the true value of the variable or construct it aims to assess. Accuracy ensures that measurements are as close as possible to the true scores or values. Accuracy is often seen as closely related to validity, as a highly accurate measure should also reflect the intended concept accurately.
Reliability
refers to the consistency and stability of a measure over time or across different conditions. A reliable measure will produce similar results when administered multiple times under similar conditions, minimising error or random variations. Reliable measures ensure that observed changes or differences in scores are due to actual changes in the construct rather than measurement inconsistencies.
Validity
is the extent to which a measure accurately captures or reflects the specific construct it intends to assess. A valid measure assesses the right construct and does so accurately. Types of validity include construct validity (does the measure capture the intended construct?), criterion validity (does it correlate with other measures of the same construct?), and cross-cultural validity (is it applicable across different cultures?).
Sensitivity to change
is the ability of a measure to detect changes in the construct being measured over time, especially in response to interventions or other significant events. A measure with high sensitivity to change can show differences or improvements, which is especially important in tracking the impact of treatments or the introduction of a new technology.
Acceptable
refers to how suitable a measure is from the participant’s perspective. An acceptable measure is easy for participants to understand, complete, and engage with, requiring minimal effort. Acceptability is crucial because if a measure is too burdensome or complex, participants may not complete it accurately or may drop out, affecting data quality.
Recruitment tips
A few points to consider when recruiting for your research study:
Defining your target population
Specify characteristics and demographics like age; type of difficulty; and experience of diagnosis to name a few. Try to aim for a representative sample. These might be people who are more likely to use the product in the future.
Reaching participants
A typical bias in recruitment is to do limited outreach. Most commonly, using social media or other internet platforms that are considered efficient and cheap.
While these allow you to advertise the study to many participants, these strategies can introduce a bias in the sample. Often those with access to technology, or those who can use the technology, tend to be part of privileged groups.
It is a good strategy to use multiple recruitment channels to overcome this potential issue of bias. Not only using social media and online forum forums, but partnering with community centers; professional organisations; charities; schools; clinics; and community groups. This can help to make your recruitment more balanced.
Planning
It is good practice to recruit in line with a plan. This ensures that you stick to a number, avoiding over or under recruiting.
Feedback
Try to seek feedback from experts by experience on your recruitment procedure as often as you can.
Mindset XR Module 18: Analysis and interpretation
Welcome to the Mindset Extended Reality (XR) Innovation Support Programme learning resources, which include three series delivered in conjunction with our expert Mindset-XR programme partners:• Medi
The Health Innovation Network use cookies to assist with Google Analytics, in order to improve the user experience of this website. We also use cookies to enable our Twitter feed. We set cookies to automatically expire after 12 months.
