Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs

Amit Baumel, Stav Edan, John M. Kane

Research output: Contribution to journalArticlepeer-review

Abstract

Trial settings that include proactive recruitment, human contact, and assessment procedures may substantially impact the way users engage with unguided e-mental health programs and the generalizability of reported findings. This study examined the impact of trial setting on user behavior by directly comparing reported user engagement in trial-based research and objective measures of real-world usage of the same unguided mental health programs. The authors conducted a systematic search for papers reporting user engagement with off-the-shelf unguided e-mental health programs. Real-world usage was obtained from a panel that presents aggregated nonpersonal information on user engagement with digital programs across the world. A total of 13 papers yielding 14 comparable usage metrics met all inclusion criteria. In three papers reporting the use of programs by lay users without any proactive trial procedures, the ratios calculated by dividing the usage reported in the paper by the usage documented within the objective dataset were 0.84, 1.05, and 1.27-suggesting a sufficient criterion validity for our examination. In studies that proactively recruited users and included pre-to post-assessment procedures (11 comparisons), the median program usage rate reported was 4.06 times higher (IQR = 4.49) than the real-world usage of the same program. Severity of clinical symptoms, in-person versus remote assessment procedures, study design, and program cost had no impact on these differences. The results suggest that trial settings have a large impact on user engagement with unguided interventions and, therefore, on the generalizability of the findings to the real world.

Original languageEnglish
Pages (from-to)1020-1033
Number of pages14
JournalTranslational Behavioral Medicine
Volume9
Issue number6
DOIs
StatePublished - 11 Oct 2019

Bibliographical note

Publisher Copyright:
© 2019 Society of Behavioral Medicine. All rights reserved.

Keywords

  • Adherence
  • Digital
  • Mental health
  • User behaviors
  • User engagement
  • eHealth

ASJC Scopus subject areas

  • Behavioral Neuroscience
  • Applied Psychology

Fingerprint

Dive into the research topics of 'Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs'. Together they form a unique fingerprint.

Cite this