Administration and Leadership, EMS Insider

Key Performance Indicators: What to Measure & Why

This is the first in a series of articles about performance measurement and improvement. Part 2, “Where Do We Start?” can be found here. Future articles will be featured in EMS Insider and will present case studies about specific EMS agencies’ improvement efforts.

A key performance indicator (KPI) is a measurable value that demonstrates how effectively an organization is achieving key objectives.

KPIs evaluate the success of an organization or of an activity in which it engages. In many cases, success is simply the repeated, periodic achievement of some level of an operational goal. Depending on the organization, the goal may be having zero product defects, achieving 10 out of 10 in customer satisfaction, or any other type of desirable organization outcome. In cases where these goals seem far off and difficult to achieve, success is defined in terms of making progress toward strategic goals.

Because of the role that KPIs play in the trajectory of an organization, it’s important to choose the right ones. Choosing the right KPIs to measure performance improvement requires a good understanding of what’s important to the organization.

An influential article focused on management goals considers the critical nature of creating objectives while also recognizing the difficulty of setting the right goals.1 This paper explained that, ideally, any organizational goal or objective should be “SMART:”

  • Specific: Target a specific area for improvement;
  • Measurable: Quantify or at least suggest an indicator of progress;
  • Agreed upon: Specify who will do it;
  • Realistic: State what results can realistically be achieved, given available resources; and
  • Time-related: Specify when the resuls can be achieved.

Key stages in an organization’s performance improvement efforts.

What’s Necessary to Ensure Improvement?

Every EMS organization is unique, with varying degrees of access to different levels of resources and manpower. Because of this, they all have different capabilities, and can expect varying standards for performance.

With this in mind, EMS organizations should measure themselves by their own, distinct goals. What would be considered successful for a well-staffed and well-funded agency might be unrealistic for a small agency with fewer resources.

Any one of the following issues can prevent organizations from committing to process improvement:

  • A lack of people with the right skills to dedicate to system performance analysis;
  • A lack of funding to purchase the right tools; and
  • A lack of access to the data necessary to make valid assessments.

High-performance organizations use measurements and analytics to allocate resources and manage quality improvement. They invest in their organizations with the goal of developing a culture that turns to metrics and data analysis for system improvement, capability and performance. These organizations usually are committed to attaining the following:

  • People with relevant knowledge and skills to analyze the data;
  • Committed investments for tools to support process improvement; and
  • Access to data with transparent availability and full cooperation of data owners.

Current Challenges in EMS

Under a contract with the U.S. Department of Homeland Security (DHS), DataTech911 conducted a study in which more than 100 EMS agencies were interviewed and surveyed. Survey results revealed the following:

  • The need to categorize KPIs (e.g., financial performance, response improvement, patient outcomes, and workforce management);
  • The need for agencies to select and customize KPIs relevant to their organization; and
  • The need for demographic stratification (i.e., wilderness, rural, suburban or urban). Based on the area covered by an agency, different KPI thresholds should be defined for each category.
  • During the interview and survey, process the following challenges and process issues were reported:
  • Data is rarely available at the correct time and in the right form and format;
  • Data collecting processes affect the results (e.g., time on scene is based on arrival on scene or patient contact);
  • Scheduled calls left in the dispatch queue for days can invalidate emergency response time information. If a scheduled call is left in the dispatcher’s queue, it can grossly affect the average dispatch time to the point where the measure has no meaning for emergency calls;
  • Mixing incident data for special events (i.e., units staged on-site) with emergency 9-1-1 calls can affect analysis. When calculating KPIs such as unit hour utilization, cost per call, or response times, mixing special events calls with emergency calls can skew the results; and
  • Most organizations don’t own all their data stores (e.g., local EMS organizations rely on county-owned computer-aided dispatch for incident information).

Other roadblocks that agencies frequently encounter include:

  • Date and time stamps: Are they accurate?
  • Software mapping issues: Are different pieces of software “speaking” to each other?
  • Element completion: Are important stages in the process being properly recorded?
  • Null/unknown values in tracking system: Are all elements of necessary data being collected? How are Null/unknown values accounted for in the overall metrics?

Where to Start?

Taking the first step to improve your organization’s performance can be daunting, and often great successes can come from taking things in a step-by-step manner.

To focus the performance improvement effort, start by taking the following steps:

  1. Identify an organization goal.
  2. Use the SMART approach to define a relevant KPI.
    1. Determine the required data and how to collect it.
    2. Collect the data and calculate the KPI.
    3. Analyze the KPI data.
  3. If the goal isn’t met, determine the root cause: What issues or obstacles stand in the way of meeting your goals?
  4. Identify and establish a process for improvement.
  5. Repeat Steps 2, 3 and 4 until the goal is achieved—or determine whether the goal is attainable given current constraints.
  6. Start again with Step 1, and identify a new a new goal to meet.

How Should I Use KPIs to Measure and Optimize?

This should be an agile, iterative process where the goals and objectives are assessed periodically and adjusted as needed.

Organizations should use current company goals (operational, personnel and financial) to define KPIs that support incremental, measurable improvement.

KPIs should be thoroughly defined in terms of what they mean, how they should be measured, and when to calculate them—and all personnel should be trained on these topics.

Present the concept of KPIs, their analysis and improvement objectives to staff in a setting where the discussion is comfortable. Transparency is key to performance improvement. Open lines of communication will ensure that the right conclusions are being made based on the data collected.

Those findings will allow for development and implementation of the appropriate corrective actions, and for another measurement cycle to begin.

As the KPIs reach acceptable levels, the next set of organizational goals and objectives should be assessed, new KPIs should be defined and the cycle should continue.

The study resulted in an agile process of assessing achievable improvement goals, measuring the performance, interpreting the level of improvement achieved and implementing change that will get closer to the final goal of providing continuous quality improvement.

References

1. Doran GT. There’s a S.M.A.R.T. way to write management’s goals and objectives. Management Review. 1981;70:35–36.

KPI Series:

Part 2: Where Do We Start?