A systematic methodological review of best practices, pitfalls, and opportunities in mixed methods research in applied clinical informatics

INTRODUCTION: Mixed methods are used to holistically understand the "what", "how", and "why" questions within a single study by integrating quantitative and qualitative methods. Although this approach has demonstrated value in other disciplines, their use in applied clinical informatics research remains largely unexplored. This methodological review characterized the types of informatics intervention studies that used mixed methods designs and the specific methods of evaluation and reporting. MATERIALS AND METHODS: On December 13, 2024, we searched Embase, MEDLINE, PubMed, CINAHL and PsycINFO for studies that reportedly used a mixed methods design to study the development or evaluation of clinical informatics interventions. We developed themes in topics studied, designs used, and reporting issues. Quality assessment was conducted using the Mixed Methods Appraisal Tool. RESULTS: We included 54 studies. Despite being described as mixed methods, 14 studies (25.9%) lacked the integration of quantitative and qualitative data. Of the remaining studies, convergent study designs were commonly used. Mixed methods were predominantly used during pre-implementation and post-implementation evaluations. Quality issues included non-representative samples and non-reporting of qualitative approaches and paradigms. DISCUSSION AND CONCLUSIONS: The use of mixed methods to study clinical informatics development and implementation is uncommon. This review confirmed that reporting and design problems found in other disciplines extend to informatics. Calls to action include a need to disseminate mixed methods guidance within the informatics community and offer mixed methods training in graduate programs. We also synthesized an initial reporting checklist from the literature. Our findings offer baseline levels to assess educational efforts.