The Mountain Model for Evidence-Based Practice Program Evaluation and Policy Analysis

To meet the recommendations of The Future of Nursing 2020-2030: Charting a Path to Achieve Health Equity, nurse leaders must have the necessary knowledge and skills to impact practice, policy, and programs.1 Although nursing has fully embraced evidence-based practice (EBP), and the literature shows evidence of movement toward sustainable practice changes in health care based on quality improvement (QI) methods, examples of nurse leadership and participation in robust policy analysis and rigorous program evaluation are limited. To address this gap and drive health care practice change, nurse-led policy and program initiatives should be informed by research and EBP (including the best available research or other evidence, clinical expertise, and population preferences). The Mountain Model for Evidence-Based Practice Quality Improvement (EBPQI) is an umbrella framework developed to merge the EBP and QI paradigms (see Figure 1).2 This model posits reliance on external research evidence (a central tenet of EBP, which is absent from QI models) and continuous monitoring for improvement and sustainability (a central tenet of QI, which is lacking in EBP).3-5 This article describes how the Mountain Model for EBPQI can be practically adapted to guide EBP program evaluation (EBPPE) and EBP policy analysis (EBPPA) initiatives, and provides examples of how the adapted models can be used in practice (see Figures 2 and 3).Figure 1.: The Mountain Model for Evidence-Based Practice Quality ImprovementFigure 2.: The Mountain Model for Evidence-Based Practice Program EvaluationFigure 3.: The Mountain Model for Evidence-Based Practice Policy AnalysisThe EBPPE and EBPPA adaptations of the Mountain Model for EBPQI retain its key components: an evidence-based foundation, critical appraisal of evidence, use of internal evidence, development of a purpose and measurable aims, and dissemination. The Mountain Model for EBPQI allows for incorporation of various EBP frameworks and QI strategies; similarly, the EBPPE and EBPPA adaptations facilitate the use of frameworks and models specific to program evaluation and policy analysis. THE MOUNTAIN MODEL FOR EBPPE Although the word “program” can have different contextual meanings (project, initiative, clinical pathway, guidelines), all imply a core set of characteristics and activities. For example, a program evaluation might include a systematic assessment of a program's processes and outcomes to enable a judgment or decision about its effectiveness, value, or other predetermined metric; however, the ultimate purpose of the evaluation will be to answer the question, “Should we keep the program as it is, change it, or end it?” To begin the program evaluation process using the Mountain Model for EBPPE, the team must identify evidence that supports the need for the program, becoming experts on the problem the program addresses and on best practices to address it. At the model's foundation level, a search of the external research evidence (the most rigorous form of evidence; however, other types of external evidence can be considered to address a specific problem) is conducted. At the next level, EBP, the evidence is critically appraised and synthesized. The internal evidence level requires an assessment of the program's current status using any data collected after its initiation. At the purpose and aims level, the nurse and their team can work with the program's developers to determine what might be measured using SMART (specific, measurable, achievable, relevant, and time-bound) aims to meet the purpose of the program evaluation. The key distinction between the Mountain Model for EBPQI and its adaptation for EBPPE is evident at the next level, program evaluation, where program evaluation methods are substituted for QI methods. Upon completion of the evaluation, the results will inform the decision to eliminate, change, or sustain the program. The pinnacle of the mountain continues to be dissemination (via, for instance, a full report to the organization; an executive summary; local, regional, national, or international presentations; publications in professional journals) to provide exemplars and guide similar programs in other settings. PROGRAM EVALUATION FRAMEWORKS USED IN HEALTH CARE The single differentiating component of the Mountain Model adapted for EBPPE is its use of program evaluation frameworks and models rather than QI methods (such as define, measure, analyze, improve, and control; or Plan–Do–Study–Act). The following are brief descriptions of some program evaluation models and frameworks that may be used. The logic model maps the causal process through which programs are expected to bring about change and produce the outcomes of interest.6 This model uses a visual representation of a causal pathway that begins with the identified problem and links program inputs (resources required) and outputs (activities/interventions and participation) to outcomes (short-, medium-, and long-term) that demonstrate impact. The RE-AIM framework evaluates programs or interventions using five dimensions: reach (who is affected, target population), efficacy (impact, outcomes, and unintended effects), adoption (assessment of the program or intervention's implementation by the organization or providers); implementation (the program or intervention's fidelity to its design); and maintenance (sustainability over time).7 This framework was developed specifically for use in public health to demonstrate population impact. Kirkpatrick's training evaluation model is used to assess training programs, often in clinical and educational settings, and has been used in professional development.8,9 Its four levels of evaluation, from simple to complex, are (1) reaction (perceptions of the participants in the training—often a measure of satisfaction), (2) learning (whether knowledge or skills have been attained from the program), (3) behavior (participants' use of knowledge and skills in practice), and (4) results (impact of the application-level behavior change on organizational or health outcomes). Levels 3 and 4 (behavior, results) are critical to evaluating whether and how projects impact health and health care. The Centers for Disease Control and Prevention Framework for Program Evaluation, published in 1999 and updated in 2024, is a comprehensive public health program evaluation guide that can be used for any health care initiative.10,11 There are six steps in the process: (1) assess context, (2) describe the program, (3) focus the evaluation questions and design, (4) gather credible evidence, (5) generate and support conclusions, and (6) act on findings. Three “cross-cutting actions”—engage collaboratively, advance equity, and learn from and use insights—must be incorporated into each step. High-quality standards for relevance and utility, rigor, independence and objectivity, transparency, and ethics should guide the evaluation. Realist, or realistic, evaluation examines which interventions are working, for whom, in what contexts, and why. It analyzes interventions through the lenses of context (the environment in which the intervention is implemented), mechanisms (the processes through which the intervention is supposed to work), and outcomes (observed effects), and explores underlying reasons for program success or failure.12 Unlike evaluations that ask only whether an intervention is effective, realist evaluation seeks to elicit understanding of the conditions under which an intervention is most likely to succeed for different groups of people. The Easy Evaluation framework originated in public health in New Zealand.13 This hybrid framework borrows from the logic model and the Centers for Disease Control and Prevention (CDC) Framework for Program Evaluation (1999 version) and has seven phases, beginning with a thorough description of the program. Phase 2 entails evaluation of a logic model of the program outcomes (if already developed) and what needs to be done to achieve them; if program outcomes have not been developed, the evaluator must create a logic model, each component of which can then be evaluated. In Phase 3, the priorities for evaluation are developed, and in Phase 4, the evaluator determines how priorities will be measured. Measurement data are collected in Phase 5, and in Phase 6, these data are interpreted and conclusions are drawn. In Phase 7, conclusions, recommendations, and lessons learned are shared and disseminated. EXEMPLAR: USING THE MOUNTAIN MODEL FOR EBPPE The administration of City Hospital wants to evaluate the hospital's six-month nursing orientation program, which has been in place for three years. The program's purpose was to increase retention of new nurses beyond two years to 75% and decrease the attrition rate to 25%, but this goal has not been achieved. When the program began, the attrition rate was 50%, but the current attrition rate is 38%. Nursing administration and human resources have identified new nurse retention as a priority problem. A nurse manager of a unit frequently affected by new nurses leaving before two years has decided to lead a team to address this issue. The nurse manager and team use the Mountain Model for EBPPE to guide their efforts. Beginning at the model's foundation level, the team searches for external evidence in order to gain expertise in the problem of new nurse retention and the most effective nursing orientation practices. Moving to the model's EBP level, the team synthesizes the evidence collected, focusing on the problem and best practice recommendations for onboarding new nurses, then compares this information with the current onboarding practices and outcomes at City Hospital (internal evidence level), which will support the team's creation of a description of the program evaluation aims and the criteria for meeting them (purpose and aims level). City Hospital's administration guides the team in describing the program evaluation's purpose as follows: “The purpose of the evaluation is to determine whether to continue the orientation program in its current state, make modifications, or stop the program.” The team will use Kirkpatrick's training evaluation model to evaluate the orientation program for new graduate nurses (program evaluation level).9 With this evaluation model, SMART aims are developed for each of its four levels of evaluation. Reaction level. Evaluation at the reaction level will answer the following question: “How did participants feel about the training?” This internal information is collected via post-orientation surveys obtained from the nursing education department and includes data going back to the time of the orientation program's introduction. These data demonstrate new nurses' level of satisfaction with their orientation experience and can be used in the evaluation. The SMART aim developed for this level is: “At least 85% of new nurses rate the orientation program as effective and supportive in helping them transition into practice.” Learning level. Evaluation at the learning level will address the knowledge and skills attained by new nurses in the orientation program, during which they must demonstrate competency in a required list of skills in simulated scenarios before starting work on hospital units. The SMART aim developed for this level is: “In each orientation cohort, 100% of nurses demonstrate competency of required skills.” Behavior level. Evaluation at the behavior level will determine whether new nurses are able to apply the knowledge and skills gained in orientation in their practice on hospital units. The SMART aim developed for this level is: “Nurse managers will be satisfied with the new nurses' preparation for practice after orientation.” Results level. This is the most important level for the organization, because the overall purpose of the orientation program is to effectively support and retain competent nurses. Two SMART aims are developed for the results level: (1) “75% of new nurses will be retained for two years, measured by comparing year-over-year retention rates for nurses who have completed the program since its inception,” and (2) “Turnover costs decrease by at least 15% over three years.” The team's next step is to conduct the evaluation initiative. For the reaction-level aim, data are analyzed from post-orientation surveys over three years, revealing that 89% of new nurses rated the program as effective in supporting their transition to practice. Similarly, for the learning-level aim, 100% of nurses achieved competency on the skills list. However, no information is available on how these nurses are functioning as they work independently on their units (per the behavior-level aim); therefore, the team decides to ask nurse managers if they have any concerns regarding the new nurses or suggestions for changing the orientation program. Upon evaluation of these data from the nurse managers, an overarching message is that they feel the new nurses need continued support beyond the six-month orientation period. City Hospital's nurse retention data from the past three years show a trend of increasing retention that has stalled over the past year (results-level aim 1). Similar findings are shown with turnover costs, suggesting that as retention increased, costs decreased, but have now reached a plateau (results-level aim 2). (See Figure 4.)Figure 4.: Retention Rate and Turnover CostsBased on this three-year evaluation, the team recommends that the nursing education department add a six-month structured mentoring program to follow orientation. The team advises that the mentors in this program will need initial training (which can serve as professional development) and that a pilot program should be implemented and continuously evaluated using the Mountain Model for EBPPE. The savings associated with improved retention, as noted in the team's evaluation of turnover cost data, reassure the administration that the orientation program is beneficial, and the team establishes that the additional costs of the mentorship program will be continuously monitored and evaluated annually in relation to the retention rate. By continuously evaluating and adapting the nursing orientation program, the organization can ensure sustained success in nurse retention (per the results-level aim 1) and professional development. As many organizations struggle with new nurse retention, the team's systematic evaluation should be disseminated so that others can undertake similar initiatives in their own organizations (the pinnacle of the Mountain Model). THE MOUNTAIN MODEL FOR EBPPA Policy analysis projects begin with the recognition of, first, a problem that could possibly be addressed with a new or revised policy or, second, a policy that should be evaluated and possibly revised or eliminated. Policy analysis often involves laws and regulations at the federal or state level; however, policies can be analyzed at the local, organizational, or unit level to significantly improve daily care practices. As experts in patient care and health, all nurses can influence and implement health policies. As with using the Mountain Model for EBPQI to address a clinical problem or the EBPPE adaptation to evaluate a program, addressing a policy problem with the Mountain Model for EBPPA requires the nurse leader and team to start at the foundational level and become experts in the policy problem by searching for external research evidence. The databases used to identify EBPPA evidence may differ from those for EBPQI or EBPPE projects, but the process remains the same. The goal of the model's foundational-level evidence search is to comprehensively describe (1) the problem that the policy is meant to address, and (2) why a policy needs to be developed, revised, or repealed based on the evidence. At the model's EBP level, the evidence must be critically appraised and synthesized. Next, a determination is made as to what internal or other data should be collected (internal evidence level). A purpose statement and SMART aims for the analysis must then be developed (purpose and aims level). The next level, policy analysis, requires the nurse leader and team to conduct an EBPPA using a policy analysis framework (this replaces the Mountain Model for EBPQI's QI level). The summation of this policy analysis will involve policy development (to address the identified problem) or recommendations for change (to a current policy). At the pinnacle of the Mountain Model for EBPPA, dissemination of the team's policy analysis process, its results, and related recommendations should occur (for example, via opinion articles or editorials, fact sheets for policymakers, presentations before committees or legislative bodies or at professional conferences, or publications in professional journals). POLICY ANALYSIS FRAMEWORKS FOR HEALTH CARE The following are some examples of frameworks that can be used at the policy analysis level of the Mountain Model for EBPPA to guide health care policy analyses: The CDC Policy Analytical Framework is a structured approach to policy analysis with three domains: problem identification, policy analysis, and strategy and policy development (including policy enactment and implementation).14 This framework is best suited for public health contexts but can work in most clinical settings. Kingdon's multiple streams framework proposes that policy change happens when three streams (problem, policy, and politics) converge, a or for The problem describes the that at any The policy describes all the to address the problem. The the current The Framework for Health and evaluation of or and in order to identify on population The framework uses seven questions on and to that may health The model is a systematic approach to public policy evaluation and using The model of four (1) and the policy (2) data, (3) the data comprehensively and evaluate and policy and (4) a policy based on evidence and EXEMPLAR: USING THE MOUNTAIN MODEL FOR EBPPA The of a health by a patient a health care organization has concerns about an and have been about and for some but now of the of are to this has public support for policy interventions to and The decides to the by a team of to how can be effectively To the team decides to use the Mountain Model for EBPPA with Kingdon's multiple streams framework the model's policy analysis level). The as by the is that in health care is a that is in and settings. At the external research evidence level of the Mountain the and team search for evidence to and describe this evidence-based to address the and from organizations as the the and Health and the Moving to the next level of the Mountain EBP, the the in using their EBP skills to the evidence on the and of health care show that in health and are most at for and that this retention, health, and patient care and to and health With this to the next level of the model, the team seeks internal evidence to describe the current state of in the organization and uses these data to aims for a policy initiative. Kingdon's these data the problem The team to a by the organization, in three nurses a at but only the The results show that many nurses work to for their These with by a nursing should be noted a not been the team have made so a priority because it is to evaluate the impact of a policy or policy change in the these the team now to the model's purpose and aims level and the overall purpose of the policy change as in the the following SMART aims for the policy to decrease by the of nurses who experience a to increase by the of nurses who report to decrease by the of to of The team is now to recommendations for policy change (the policy and work the to the team must determine the and organizational that will influence the and of At the policy analysis level of the Mountain using Kingdon's multiple streams it is evident that a is for the focusing (the the policy analysis, and the policy, and streams are The team has the evidence to the organization for their support and a policy By the policy from and cost as as a the team the resources and to implement an evidence-based policy on units where are at for The policy with practices and includes organizational and a new for The and team may related to costs and about the program's on but they can these concerns with data from external and internal evidence supporting improved retention, and patient care can beyond their work and the to a professional organization or to their have that organizations make their reached the of the Mountain Model the team should their work in this as the may other organizations and is with to other settings. The team should the results to their organization, at conferences, and through The Mountain Model for EBPQI can be adapted for program evaluation and policy analysis The Mountain Model understanding of a systematic process of searching and using supporting evidence to achieve change and the outcomes of the The program evaluation and policy analysis methods in this article are not a comprehensive are they in however, they are to increase that multiple frameworks and models that can be incorporated into the Mountain adaptations to the resources in the nurse Program evaluation and policy analysis informed by evidence can add value, support use of and improve the quality of care. are in an to lead EBPPE and EBPPA initiatives, as as EBPQI that completion of a may lead EBPPE and EBPPA initiatives as an to EBPQI Nursing education programs are to related to program evaluation and policy analysis in their and related in order to support projects in that not or As nurses can and education and with experts and to the knowledge and skills to undertake program evaluation and policy analysis (the models and frameworks in this article can serve as a starting related to the of EBPPE, and EBPPA are and can be used to address most that nurses may in practice. The Mountain Model a robust and framework to guide nurses through EBPQI initiatives, program and It a foundation of research and evidence with to ensure systematic for in health care and