So Hyeon Bang, Soojeong Han, Meghan Reading Turchioe, Melani Ellison, Stacey Dai, Mary Beth Happ, David Russell, Ruth Masterson Creber
LLMs are best positioned as analytic partners rather than autonomous coders. Transparent workflows with human-in-the-loop validation are essential for responsible AI integration in health and biomedical informatics.
OBJECTIVE: To develop and systematically compare a human-led and LLM-assisted hybrid deductive-inductive workflow for qualitative analyses.
MATERIALS AND METHODS: We analyzed 122 transcripts (n = 61 research clinical consultations; n = 61 reflexive interviews) from a video ethnography study of patients with heart failure. Human-led thematic analysis used Dedoose software, and LLM-based analysis was conducted using ChatGPT Edu (GPT-5.2; OpenAI) with an eleven-prompt protocol. Both applied a hybrid deductive-inductive approach. The research team compared outputs across 63 human-LLM theme pairs using human consensus and LLM-based evaluation, integrated themes into a final framework, and manually verified quotation fidelity against the original transcripts.
RESULTS: Human-led and LLM-generated analyses produced complementary cross-cutting themes (7 human; 9 LLM), all judged valid and integrated into 14 final themes across three domains. Thematic overlap was moderate to substantial (Hit Rate 1.00; Jaccard 0.44-0.51). Robustness testing across three runs revealed recurrence of five core concepts alongside variability in theme labels and counts. Quotation fidelity showed 68% verbatim, 20% paraphrased, 6% partial and 3% full hallucinations, and 3% truncated excerpts; verbatim quotations did not always clearly support their assigned themes.
DISCUSSION: LLM-assisted analysis is feasible for large-scale qualitative health research within a HIPAA-compliant environment using an adaptable eleven-prompt protocol. Human oversight remained essential for contextual interpretation, quotation verification, and assessment of theme-quotation support.
CONCLUSIONS: LLMs are best positioned as analytic partners rather than autonomous coders. Transparent workflows with human-in-the-loop validation are essential for responsible AI integration in health and biomedical informatics.