Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs

Trial settings that include proactive recruitment, human contact, and assessment procedures may substantially impact the way users engage with unguided e-mental health programs and the generalizability of reported findings. This study examined the impact of trial setting on user behavior by directly comparing reported user engagement in trial-based research and objective measures of real-world usage of the same unguided mental health programs. The authors conducted a systematic search for papers reporting user engagement with off-the-shelf unguided e-mental health programs. Real-world usage was obtained from a panel that presents aggregated nonpersonal information on user engagement with digital programs across the world. A total of 13 papers yielding 14 comparable usage metrics met all inclusion criteria. In three papers reporting the use of programs by lay users without any proactive trial procedures, the ratios calculated by dividing the usage reported in the paper by the usage documented within the objective dataset were 0.84, 1.05, and 1.27-suggesting a sufficient criterion validity for our examination. In studies that proactively recruited users and included pre-to post-assessment procedures (11 comparisons), the median program usage rate reported was 4.06 times higher (IQR = 4.49) than the real-world usage of the same program. Severity of clinical symptoms, in-person versus remote assessment procedures, study design, and program cost had no impact on these differences. The results suggest that trial settings have a large impact on user engagement with unguided interventions and, therefore, on the generalizability of the findings to the real world.