Evaluating Large Language Models as Post Hoc Explainability Interfaces for Credit Risk Models

ABSTRACT Large language models (LLMs) are increasingly considered as potential interfaces for communicating model‐based explanations in credit risk decision‐support settings. This study evaluates whether LLMs can serve as post hoc explainability tool for credit risk models, focusing on their ability to preserve supplied feature‐importance rankings and autonomously generate feature‐importance rankings. Using a LendingClub dataset, we compare LLM outputs with SHAP and coefficient‐based attributions. Results indicate that LLMs reliably reproduce reference rankings under controlled prompts but show limited alignment when generating explanations autonomously. These findings suggest that LLMs are best deployed as narrative interfaces rather than substitutes for formal attribution methods in credit risk governance. The paper is organized as follows. Section 1 introduces the research problem. Section 2 reviews the related literature on explainable AI and the use of LLMs in financial applications. Section 3 describes the experimental design, including data processing, baseline model training, prompt engineering strategies and evaluation metrics. Section 4 presents the empirical results for both research questions. Section 5 concludes with a discussion of implications and directions for future research.