The value of performance measures is pretty clear: Once we see how we’re doing it makes it much easier to know how to get better. And the only way to know if we’re getting better is to measure our processes and the outcomes we’re trying to achieve. But creating a good performance measure isn’t as easy as it sounds.
Types of Measures
There are several different types of measures. To do our jobs, we need to have the people, equipment, supplies, vehicles, computers, software and training in place. When we track or count these tangible things, the infrastructure that’s needed to get the desired outcome is called “structure measures.” These measures are important for budgeting and allocating resources, but aren’t always clearly linked to outcomes.
Once the right structure is in place, we need to take action and do something with all those resources. When we measure that action, those things we do, they’re known as “process measures.” In EMS, process measures often look at whether or not we performed an intervention appropriately. Even these can be tricky, as in the past we’ve often measured processes that weren’t directly tied to outcomes.
Outcome measures look at what matters most to our patients and communities. The best examples are pretty simple to understand but often harder to actually calculate or access the right data for. People want to know if we make communities healthier-if we save lives, reduce suffering and improve recovery.
Think of that 14-ounce titanium milled, curved, hickory-handled hammer on sale at your local hardware store. I assume, like most of you, that when I buy a hammer, I’m buying a hammer because I need to put a nail in the wall to hang a portrait. I’m not buying it just because I want to own a hammer.
A structural measure would simply look at whether you own the hammer and nails. A process measure might look at whether you put the nail in the right place, hit it at the proper speed and to the proper depth. The outcome measure would actually look at the final product-is the picture on the wall? Did it stay up there for more than 10 seconds?
In these ways we’re using performance measures to look at what’s happened. Do we have the right structure in place to support the right processes that will achieve the right outcome?
There’s one other type of measure, one that we don’t use as much but can be critically important: the balancing measure. Balancing measures are used to make sure we don’t have any unforeseen and negative consequences. After hanging the portrait, our outcome is achieved-but in the process, were all the other pictures on the wall knocked off? Or did you fracture your thumb with the hammer?
Identify the Desired Outcome
When creating a good performance measure, you should start with the end in mind and identify the desired outcome. Imagine it from the patient’s perspective: What’s the best outcome a patient should expect? While there are times that our outcome is related to EMS provider safety or satisfaction, in most cases an outcome that benefits a provider will ultimately benefit the patient as well.
The EMS Compass process places a lot of emphasis on simply choosing the desired outcomes and processes to measure, because it’s a key step in performance improvement. EMS history is filled with examples of measuring things that might not be tied to patient-centered, evidence-based outcomes, at times with negatives consequences. For example, there may have been a time when systems measured whether they applied pneumatic anti-shock garments (PASGs). But it turned out that PASGs may have done more harm than good for bleeding patients, so measuring their application evaluated compliance with a protocol, but not a patient outcome.
But measures also need to be applicable in the real world, so they’re not developed solely by the researchers and scientists. The first step in the EMS Compass process is an open “call for measures” that’s widely announced and publicized through multiple publications, websites and social media to reach as many people as possible. Members of the EMS community and the public then have several weeks to submit their performance measure ideas through the EMS Compass website. Even after all the measures are submitted, public feedback is still solicited throughout the process. For example, the first EMS Compass call for measures was followed by a series of webinars where anyone could learn more about the measures, ask questions and propose ideas.
After the call for measures and further feedback from stakeholders, the EMS Compass Project Execution Group, which includes the chairs of the initiative’s working groups, works together to refine the vast array of proposed measures. This summary is then provided to the Steering Committee to prioritize and provide overall guidance. Measures are grouped into families aligned with clinical conditions or other general topics. For example, a stroke family of measures includes several process and outcome measures related to stroke, all of which are important in improving overall stroke care.
The next step is vital to the success of any measure but is most obvious for the design of clinical performance measures. In order to ensure that measures encourage and improve evidence-based care, a talented group of state and local medical directors, paramedics and researchers review the available published scientific literature. In reading the science, the Evidence Review Group considers the size of the study, how scientifically valid the methodology was, the outcomes that were measured and the reported results, as well as other relevant details.
Fortunately, some of the legwork for evidence review has already been done by groups such as the American Heart Association or through the national Prehospital Evidence-based Guidelines (EBG) Consortium. Although the Evidence Review Group still closely looks at these studies and their methodology, and examined more recent literature to see if any studies had shed new light on the topics, thorough and thoughtful grading of evidence had already been performed. In the future, one can imagine an even closer marriage of the two processes, as evidence-review is simultaneously used to develop EBGs and related performance measures.
Building the Measure
Once it’s confirmed that a measure is supported by medical research and best practices, the next step is to actually design the measure. It’s one thing to say we want to measure whether or not EMS providers appropriately identify and assess stroke patients-it’s another to figure out exactly how to measure that. And that’s one of the main reasons for the existence of EMS Compass: to develop measures that can be used consistently by any EMS agency so they can work with each other to improve across borders.
The EMS Compass Measurement Design Group, whose members represent a wide range of EMS organizations and providers, including Fire-based, private, public, ground and air services located in rural, suburban, and urban areas of the United States, uses the graded evidence to make the measures. They discuss what the evidence shows so that they can create the first drafts of measures in “plain English.” In other words, they talk about which patients would be included in the measure, how to define what gets counted, and more. By discussing the measures in simple terms, they’re able to test the concepts in an attempt to make sure that they account for all the different types of EMS systems, geographies, levels of service and types of healthcare systems.
Once the members of the Measurement Design Group are satisfied they have a good first draft, they share it with the Technology Developers Group comprised of software architects, developers and data experts who represent the majority of all ePCR systems used to collect prehospital data in the U.S. They break apart the measure into pieces to help identify specific data elements to use.
Our profession has a huge advantage that makes EMS Compass different than performance measure projects in other areas of healthcare. The National EMS Information System (NEMSIS) is a single nationwide standard for how EMS data is collected and identified. Nearly every patient encounter in the U.S. is documented using the same data definitions, removing any significant roadblocks to using EMS Compass measures or combining or comparing data from one patient or system to another.
At this stage, the measures look like a foreign language, but each of the letters and numbers refers to a specific checkbox or field in the ePCR, allowing a software developer or EMS agency to easily determine how to calculate the measure.
After identifying the data elements to use, the technology developers do extensive testing to verify the data is being collected consistently across the country and that the measure is actually calculating what it’s supposed to calculate.
To make sure the measures work using actual EMS data collected by providers in the field, two software vendors have contributed entire datasets of tens of thousands of records of prehospital data and linked hospital outcome data. This enables the workgroup to test and tweak their draft measure definitions using real-world data. It takes several revisions before all of the complications are identified and fixed and the measures are ready for public testing and feedback.
When a measure is released for public comment, the goal is to get feedback on both the technical aspects of the measure as well as the clinical and operational implications of using the measure. How will that measure impact patients, providers, services or the community?
This is one of the most important parts of creating performance measures, and it doesn’t end after the measures are finalized and approved by the EMS Compass Steering Committee. Performance measures must continually be evaluated, as new evidence changes clinical care, new data sources become available, or unforeseen side effects of the measures are discovered. Sometimes the creation of a measure leads to the realization that information isn’t being collected, or is being collected but not in the best way.
Conclusion
The development of evidence-based care, the creation of performance measures, and the collection and analysis of data are all part of a fluid cycle that must frequently adapt and evolve. That’s why EMS Compass has focused on developing not only performance measurements for EMS, but also a performance measures development process, which in many ways is more critical than the measures themselves. Whether it’s used by individual organizations or the entire EMS community, the measurement development process can live on, making EMS Compass an ongoing collaborative initiative.
