Beyond Accuracy: Safety-Centered guidelines for the evaluation of LLM-based therapy recommendation systems for chronic multimorbidity patients

Background With the advance of large language models (LLMs), researchers are using them to develop medical applications, including diagnosis decision support. However, research on LLM-based therapy recommendation for multimorbidity remains relatively limited and guidelines for their evaluation is lacking. Objective The objective of this Special Communication is to provide medical informatics researchers with guidelines for proper evaluation of LLM-based therapy recommendation systems for multimorbidity management. Methods We developed the guidelines based on a review of the state of the art methods and our experience in using LLMs for therapy recommendations for multi-morbidity disease management and evaluating them quantitatively on clinical data sets and qualitatively on benchmark case studies. Our experience improved upon state of the art metrics. We arranged our recommendations, arranged into several categories. Results A merely technical quantitative evaluation that shows promising results may overlook crucial patient safety issues, which are only discovered in rigorous qualitative evaluation. We share recommendations for evaluation studies that rely on both quantitative and qualitative evaluation and present novel evaluation metrics that are important for therapy decision-support in the context of chronic multimorbidity patients. Discussion We further discuss considerations for the appropriateness of freely-available datasets and of existing evaluation metrics, and provide suggestions for the operationalization of the proposed recommendations in clinical settings.