• Article
  • August 30, 2019

Using the EBP Model to Solve a Problem

How a team from the ANPD Education Committee studied measuring return on investment (ROI) for professional development activities

By: Kaitie Marolf

Article 1 of the "Return on Investment: Calculating the Economic Impact of Educational Interventions" series.

A group of nursing professional development professionals on the Education Committee for the Association of Nursing Professional Development (ANPD) formed a subcommittee specific to return on investment (ROI).  They were challenged with providing the membership with education on ROI and the financial value of professional development activities.

They decided to use the 7-step Melnyk and Fineout-Overholt Evidence-Based Practice (EBP) model to guide their exploration of the topic. In a webinar hosted by CHA in January 2019, Debra Liebig, MLA, BSN, RN-BC, program manager in the Accreditation and Readiness Department at Children’s Mercy Hospital Kansas City, walked listeners through that process. View the webinar recording (CHA login required). 

Step 0: Cultivate a Spirit of Inquiry

This first step involves gathering a group of people who are interested in and excited to explore the topic at hand.

The ANPD sub-committee had the initial interest to look into as many aspects of their question as possible. 

Step 1: Ask Clinical Questions in the PICOT Format

Develop a clinical question to guide your search.

The team used the PICOT formula for writing a research question.

P

Population

Identifies the subjects of the study

I

Intervention of Interest

The main intervention/treatment of interest

C

Comparison Intervention

An intervention/treatment to compare to the intervention of interest

O

Outcome(s)

The result that will be measured

T

Time

Duration of the data collection process

 

Their resulting question was: “In healthcare organizations (P), how does calculation of ROI for professional development activities (I), compared to nonfinancial evaluation of programs (C), affect decisions about resource allocation to professional development activities (O)?”

Step 2: Search for the Best Evidence

Using keywords and reliable databases, in this step, gather what has already been written on your topic and sift through it.

They began to look for the answer to this question with a review of existing literature.

  • They searched for keywords in three groupings, including:
    • Professional development, staff development, nurse educator, education, training
    • Return on investment, benefit-cost analysis, financial impact
    • Program evaluation, educational outcome, measurement, impact, resource allocation
  • Using these databases:
    • Medline
    • CINAHL
    • ERIC

“We struggled with this, though, because there is really no consistency in the way that the concepts are reported in the literature,” Liebig said. “Many times, we found ourselves doing more literature searching as we read the articles.”

Out of 140 articles that the team read, which had been published in the last ten years (2004-2014) and included pre- and post-data or a financial impact, they found 12 studies that met their criteria. Those 12 were referenced in their first two award-winning articles. They found another four that had been published between 2013 and 2018, which they referenced in their third article.

“We hope that you guys see how small those numbers are in comparison to the numbers that we started with,” Liebig said. “We identified that people were not publishing cost of implementing education programs, which would lead to practice changes, so we knew that we had a gap.”

Step 3: Critically Appraise the Evidence 

This step includes evaluating the validity, reliability, and applicability of the study to solve the problem or answer the question at hand.

Outcomes of the studies found in existing literature ranged vastly, including attrition, lost workdays, injury rates, and job satisfaction. The investigators were monitoring as many as eight outcomes per study.

“When you think of the economic impact of your practice change, consider a wide range of outcomes,” Liebig said. “The change at least in part could affect multiple outcomes.  We highly encourage you to watch them all.”

Step 4: Integrate Evidence with Expertise and Patient Preferences and Values

Taking action with the newly gained understanding of your topic comes in here. In clinical settings, the patient’s preferences, health, and demographics should be considered before implementation.

The team put their knowledge into action in several ways:

  • Writing their three-article series
  • Presenting webinars with ANPD, CHA, the Center of Transdisciplinary Evidence-based Practice, and The FULD Institute for EBP
  • Cathleen Opperman, DNP, RN, NEA-BC, CPN, nursing professional development specialist at Nationwide Children’s Hospital, writing chapter 18 in the ANPD CORE Curriculum on “Measuring Impact at the Organizational Level”
  • Hosting half-day workshops at ANPD Conventions
  • Presenting at the 2018 ANA Quality and Innovation Conference

Throughout the process, they were also nursing another idea.

“In the meantime, intrigued by all of this, our group dreamed about an app that we could enter the data and it would spit out all the calculations,” Liebig said. “Through this app, you could make expense adjustments and outcome goals to create more viable economic impacts of your programs. We created it, and now access to it is available to ANPD Members for a nominal fee.”

Step 5: Evaluate Outcomes of Practice Decisions or Changes Based on Evidence

This step calls for evaluating the outcomes of the interventions that you put into place.

The team considered more publications with economic impact reported and increased comfort calculating financial impact for professional development activities to be indicators of the outcome of their work but have not gathered formal results yet.

“Maybe the measure of impact with this project will be another search of the literature in a few years to find many more studies including ROI and economic assessment,” Liebig said. “Maybe [it] will be incorporating economic or financial assessment as a competency for all Educators and Leadership.”

Step 6: Disseminate EBP Results

Getting the word out can greatly increase the impact of your study.

The articles, webinars, workshops, Core Curriculum, and the program to do the calculations were a combination of the group’s Step 4 implementation and Step 6 dissemination efforts.

“We think we are doing a rather thorough job with the disseminating step,” Liebig said.

 

NOTE: Stay tuned for the next article in the series:  “Measurable Outcomes of Educational Interventions”