Our Insights team use a range of different specialist approaches to understand and measure the success of innovations. In this blog, Senior Data Analyst Efejiro Ashano discusses recent work using system dynamics modelling to develop our understanding of the long-term implications of innovations and map the “ripple effect” that changes can have on outcomes.
Evaluation is a critical element of innovation. Without understanding how new technologies or models of care perform we cannot make judgements about whether patients actually derive benefits from the changes we are making, or whether they help services deliver care more effectively or efficiently.
One of the greatest challenges in evaluating healthcare innovation is the complexity of the world we work in. Our healthcare system is made up of many different services who may all play a role in managing or treating an illness, and how patients interact with these services is also complicated.
This poses a risk that evaluations focus too much on the “problem right in front of our face” and fail to provide insight into the wider implications of a new model of care or innovation.
Take, for example, a hypothetical innovation designed to spot patients ready for discharge from hospital more quickly. A superficial evaluation might focus on the innovation’s ability to identify these types of patients more quickly, and judge success on reduced length of hospital stays and the money saved by discharging patients from the service.
However, this type of evaluation might not capture the full picture of how the new innovation had impacted the system as a whole. Higher volumes of early discharges might mean more care being delivered in the community; without an expansion in community nursing capacity this could result in difficulties delivering high-quality care or monitoring discharged patients for deterioration. In turn, this could mean that a significant proportion of patients discharged earlier were readmitted or needed to access urgent care services – potentially offsetting any benefit from the innovation.
The role of system dynamics in evaluation
“All models are wrong, but some are useful”
– George Box
This famous quote from British statistician George Box neatly captures the fundamental objective of system dynamics evaluation. The real world is infinitely complex and impossible to model; system dynamics is intended to bridge the insight gap between evaluations conducted in isolation and this infinite complexity.
To achieve this objective, system dynamics uses modelling to look at how changes – such as policy shifts, different models of care, or the introduction of innovative technologies – could play out over a period of time when we account for interactions between a set number of other factors or processes. This modelling is usually built around mathematical algorithms developed to predict behaviours related to a particular scenario – such as the adoption of innovation.
Using system dynamics to model the impact and sustainability of a change to NHS 111 services
Recently, we worked with NHS England (London region) on a project looking at how the NHS 111 telephone service could be optimised to help service users access help for dental conditions more effectively.
Specifically, we wanted to look at how changing parts of the automated signposting messaging within the service could encourage more users to access a digital triaging tool, which would allow them to find the information they were looking for more quickly than waiting to speak to an advisor.
In this project, we did some initial testing of new messaging – which seemed to show that it increased the uptake of the digital tool as intended. However, we did not know whether these changes were sustainable, or whether there were potential unintended consequents of the changes, such as negative impacts on health inequalities.
To help us understand these issues, we used a model called the Bass Diffusion Model, which is a structural framework intended to model adoption.
In very simple terms, the Bass Diffusion Model was used to provide a 10-year view of how uptake was likely to be impacted by the factors we had control over (changing the messaging) compared to the factors we could not control which would organically shift uptake over time (such as re-entry to the NHS111 service or people hearing about the digital service’s benefits through other channels). The model’s full parameters were designed empirically using data taking into account factors such as seasonality and the characteristics of London’s population. We also included two age bands to model whether uptake of the tool would result in health inequalities related to age – in many cases digital tools see lower adoption in older users.
The insights we gained from the model were significant:
- The impact of our messaging changes were highly impactful, accounting to more than 90% of the estimated improvements in uptake over time;
- Positive feedback on the service and a “word of mouth” effect was likely to act as a strengthening factor for uptake;
- Uptake was unaffected by demand changes (e.g. when we modelled the introduction of similar functionality through the NHS App);
- Changing the messaging led to a reduction in age-related adoption gaps initially, although longer-term these age gaps may reappear.
On the basis of this modelling, NHS 111 has now rolled out the messaging changes across all relevant pathways – helping tens of thousands of users access help and guidance more effectively, and significantly reducing pressures on the telephone service.
Integrating system dynamics modelling into healthcare
At a time when the system is going through significant changes in response to policies such as the 10 Year Health Plan, it is critical that we do everything we can to understand the full impact of innovations or emerging models of care.
System dynamics modelling is not a “silver bullet” providing perfect intelligence, but when it is combined with robust real-world evidence, it can uncover important considerations which may influence commissioning or implementation decisions.
While no model can perfectly capture the complexity of the real world, system dynamics modelling offers a powerful tool to support better decisions, reduce uncertainty and maximise the likelihood that innovation delivers meaningful and lasting improvements for patients, populations and services.
Find out more
Interested in finding out more about this work?
Get in touch with the project team.