Using real-world data to measure impact on patient and system outcomes for a digital health technology
Context
Around a third of adults are thought to have problems sleeping each week and up to one in ten people meet criteria for insomnia disorder. Often insomnia is short term and people can treat themselves by changing their sleep habits (known as sleep hygiene) or with over-the-counter remedies.
If insomnia becomes a longstanding problem, GPs can give further advice on sleep hygiene, referral for cognitive behavioural therapy for insomnia (CBT-I), or prescribe hypnotic drugs. However, drug treatment is used less often than in the past and only for short periods because of concerns about side effects and dependence.
Sleepio is a digital health technology and sleep improvement programme designed to support people to self-manage their insomnia. It is accessed mainly through its website or iOS app and integrates sleep data from wearable devices such as apple watches and Fitbits. The programme is structured around a sleep test, weekly interactive cognitive behavioural therapy sessions and regular sleep diary entries.
The final NICE guidance was published in May 2022, which recommended Sleepio as an option for treating insomnia and insomnia symptoms in primary care for people who would otherwise be offered sleep hygiene or sleeping pills.
Key evidence
The evidence for Sleepio included 12 randomised controlled trials and 8 non-randomised studies. Randomised controlled trial evidence was supplemented by real-world evidence (RWE) studies in local NHS settings.
Of the non-randomised RWE presented, we focus on two studies which display elements of high-quality study design and reporting, as outlined in NICE’s real-world evidence framework. These studies considered clinical outcomes for patients (Stott et al. 2021) and health system outcomes (Sampson et al. 2022).
Real-world evidence for patient outcomes
Stott et al. 2021 was a cohort study using data from the Improving Access to Psychological Therapies (IAPT) audit in Buckinghamshire, UK. In this study, all individuals meeting sleep criteria were offered Sleepio.
The study compared outcomes for people who chose to use Sleepio with those who did not use Sleepio using propensity score matching. This method aims to select a comparison group with characteristics similar to the treatment group thereby minimising certain biases that are common in non-randomised studies.
The study measured patient-reported outcomes at 1 year including the PHQ-9 for general mood and the GAD-7 for anxiety.
What was done well?
The report presented a participant attrition flow diagram describing the number of people:
- meeting eligibility criteria
- offered the intervention
- completing follow-up
- declined to participate
The report also clearly listed the variables used to estimate the propensity scores and described the matching procedure. This included reporting both the absolute values of each variable and standardised differences before and after matching (Table 1, below). For example, the baseline measures of PHQ-9, GAD-7 and sleep items were shown to have smaller differences across comparison groups after matching. This means the likelihood of producing results biased by these baseline differences in participants should be reduced.
NICE’s real-world evidence framework states researchers should justify the statistical methods used and report these clearly enough to enable a RWE study to be reproduced by an independent researcher.
Table 1: patient characteristics before and after propensity score matching
Reference:
Stott R, Pimm J, Emsley R, et al. (2021) Does adjunctive digital CBT for insomnia improve clinical outcomes in an improving access to psychological therapies service? Behaviour research and therapy. 1;144:103922.
Table 1: patient characteristics before and after propensity score matching
|
Parameter |
Sleepio group |
Unmatched controls |
Matched controls |
Standardised difference with unmatched controls |
Standardised difference with matched controls |
|---|---|---|---|---|---|
|
Age (years) |
40.46 |
40.69 |
40.73 |
-0.015 |
-0.017 |
|
Gender (female) |
66.5% |
66.6% |
69.0% |
-0.004 |
0.053 |
|
Diagnosis - depression |
43.9% |
41.7% |
42.0% |
0.045 |
0.039 |
|
Diagnosis - anxiety |
52.2% |
53.7% |
54.3% |
-0.032 |
-0.043 |
|
Diagnosis - other |
3.9% |
4.6% |
3.7% |
-0.032 |
0.01 |
|
PHQ-9 |
15.20 |
14.13 |
15.21 |
0.184 |
-0.001 |
|
GAD-7 |
13.52 |
12.91 |
13.66 |
0.123 |
-0.03 |
|
Sleep item |
2.43 |
1.94 |
2.44 |
0.567 |
0.038 |
Reference:
Stott R, Pimm J, Emsley R, et al. (2021) Does adjunctive digital CBT for insomnia improve clinical outcomes in an improving access to psychological therapies service? Behaviour research and therapy. 1;144:103922.
Real-world evidence for health system outcomes
Sampson et al. 2022. used an interrupted time series study design. This study compared trends in outcomes before and after a population-wide introduction of Sleepio across the Thames Valley region of England.
Primary care data were collected from 9 GP practices through the EMIS system, which supports electronic patient records for general practices across the UK. Outcomes included change in total primary care costs, such as from general practice contact and prescriptions, after the rollout of Sleepio.
The study controlled for a range of observed patient level characteristics and for seasonality to account for differences in primary care activity across the calendar year (see Table 2).
What was done well?
A clear description of study design and statistical models was provided, and the rationale for them. For example, it was demonstrated that primary care costs tended to increase gradually over time and that adjustment for seasonal variation in costs was critical to assessing the effectiveness of Sleepio.
The study also performed sensitivity analyses to consider the impact of adjusting for different time and patient-level characteristics (Table 2). NICE’s real-world evidence framework promotes the use of sensitivity analysis to assess the robustness of study results to key risks of bias and uncertain analytical decisions.
The study estimated the reduction in primary care costs associated with Sleepio rollout by user, region, and across England. These results were used to inform the committee’s estimate of cost savings compared to treatment as usual.
Table 2: presenting results across different statistical models for sensitivity analysis
*Indicates statistical significance (p<0.001)
Reference
Sampson C, Bell E, Cole A et al. (2022) Digital cognitive behavioural therapy for insomnia and primary care costs in England: an interrupted time series analysis. BJGP open. Mar 9.
Table 2: presenting results across different statistical models for sensitivity analysis
|
Category |
Preferred model |
Model 2 |
Model 3 |
Model 4 |
Model 5 |
|---|---|---|---|---|---|
|
Time |
0.002* |
0.002* |
0.002* |
0.001* |
0.004 |
|
Intervention |
-0.038* |
-0.036* |
-0.033* |
-0.033* |
0.263* |
|
Post |
-0.002* |
-0.002* |
-0.002* |
-0.000 |
-0.003 |
|
Seasonal adjustment |
Yes |
Yes |
Yes |
No |
No |
|
Age bands |
Yes |
Yes |
No |
No |
No |
|
Practice random effects |
Yes |
Yes |
Yes |
Yes |
No |
*Indicates statistical significance (p<0.001)
Reference
Sampson C, Bell E, Cole A et al. (2022) Digital cognitive behavioural therapy for insomnia and primary care costs in England: an interrupted time series analysis. BJGP open. Mar 9.
What else could have been done?
These studies demonstrate the potential for real-world evidence to influence committee decision making by complimenting trial findings and informing economic models. Developers of these real-world evidence studies could have further improved the transparency of their evidence to reviewers by providing:
- Access to a study protocol prepared in advance of study conduct.
- A more detailed assessment of data suitability including provenance, quality, and relevance.
NICE’s real-world evidence framework describes best practices for planning, conducting and reporting real-world evidence studies. It also provides case studies and tools, for example, to help developers ensure key aspects of data suitability are reported.
Built with Shorthand