How many times have you heard a new EMT ask an experienced member of the department why something was done a particular way? Often, the response is, “Well, that is the way we have always done it.” Or, how many times have you heard some EMTs talking about a particular EMS practice, such as MAST trousers, when one says, “MAST trousers work. I have seen many patients saved who would have died had we not used the MAST.”
Such a story is common and referred to as an anecdote. The anecdote typically draws conclusions from cases that support the conclusion often while ignoring cases that might tend to undermine the conclusions. Anecdotal stories are often a statement of faith. Now, don’t get me wrong, people who tell anecdotes do not generally intend to deceive. We each come to work with varied beliefs and practices. That is human nature. Thus, two people can look at the same thing and come away with different conclusions.
For example, those who believe in CISM tend to remember the cases where they believe “it worked” and those that don’t tend to remember the cases where they believe “it didn’t work.” Who is right? Who knows? That is why anecdotes have no place in science, medicine or EMS.
To prove that something really works, such as an EMS practice, it must be independently verified. That is, it must be tested so that our personal biases and ingrained beliefs do not come into play. Thus, we use the scientific method. Science, when properly followed, eliminates bias. Astronomer Carl Sagan once wrote, “Science is a way of thinking much more than it is a body of knowledge.”
Much of the current EMS in practice is based upon anecdote primarily because a body of scientific evidence did not exist when EMS was developed more than 30 years ago. Now, we are starting to see bona fide scientific EMS research. Normally, when such research occurs, we change our practices based upon the research. Sometimes we don’t.
I recently had the opportunity to debate my good friend Mike Taigman on the practice of system status management (SSM). However, the problem was the conference was for those who operate the so-called “high-performance” EMS systems. As you can imagine, I felt like I was debating evolution at a Pentecostal convention.
After the talk I had several people come up to me and say, “Dr. Bledsoe, you had guts to come and debate in front of this crowd.” Then, invariably, they would drift into a story or stories about how they “know” that SSM works. Many had the ultimate response, which they used to end the conversation. They would state, “Well, you can have your science and your statistics.
All I know is that I have seen it work and nothing you can show me will convince me otherwise.” Not only do they resort to an anecdote, they close their mind to science. Novelist Gerald White Johnson once wrote, “The closed mind, if closed long enough, can be opened by nothing short of dynamite.”
Let’s look at some of the fundamental practices of so-called “high performance” EMS systems. First, they tend to use SSM. Second, they rely heavily of Medical Priority Dispatch (MPD). Third, they have strictly adhered to target response time (usually eight minutes).
Fourth, they try and maximize unit hour utilization (UHU). All EMS services try to get the most out of their units so I won’t quarrel with the UHU component. But, let’s look at the science that has evolved concerning the other three: SSM, MPD and response times. Soon you will clearly see that “high performance” EMS is built upon a foundation of sand.
The concept of SSM was introduced by Jack Stout in a 1983 issue of JEMS. Stout theorized that EMS demand followed a weekly cyclical pattern and using 20 weeks of call data would allow dispatchers to determine, with some degree of validity never defined, where and when EMS calls would occur. One of the tenets of SSM was that fixed stations were not necessary and a “fluid deployment model” would improve response times.
The idea seemed good on paper and several systems, especially those where Stout and his company, The Fourth Party, served as consultants, adopted the practice of SSM. However, there was not a single shred of scientific evidence that SSM worked. From a layperson’s standpoint, it seemed intuitive.
From a critical thinking standpoint, it seemed impossible. A non-scientific study, published in JEMS in 1986 found that using SSM in Tulsa, Okla., decreased response times from six minutes 46 seconds to six minutes nine seconds. But, miles traveled increased by 19% and maintenance costs increased by 38%. Everybody assumed that a decreased response time correlates to lives saved. But does it? Now, for more than 20 years, SSM has been practiced by some systems despite the lack of a single scientific study supporting the practice.
Finally, after I wrote an article on EMS myths in EMS Magazine several years ago, some of the proponents of SSM undertook a study. Although only the abstract has been published, the results were clear. SSM was unable to estimate exact call volumes just trends (peaks and troughs) in call volume. Using three different measurements they found that SSM predictions were accurate (+/- one call) only 10-19% of the time depending upon the measurement used.
Call volumes were overestimated between 74-80% of the time. None of the data reached statitisical validity. Thus, the findings were no different than simply flipping a coin. In science there is a tool called Occam’s razor that basically states when there are two possible explanations, chose the simpler of the two. Basically, it is clear that the ability to find peak and trough trends can be achieved by much simpler means.
Now, let’s look at MPD. Like SSM, MPD seemed intuitive. Using medically-approved screening protocols, dispatchers could determine what level of ambulance (ALS or BLS) to dispatch and the priority of the response. What began as a cottage industry has taken off with computerized prompts, MPD certification and so on. But, where was the science behind MPD? Well, there wasn’t any.
In recent years, researchers looked into the scientific evidence behind MPD and wrote, “There is very little evidence to support the effect of the prioritization of emergency ambulances on patient outcome.” When researchers started looking at individual protocols for SSM, they found very little correlation between patient condition and the protocol.
When looking at non-traumatic abdominal pain, researchers found that out of 186 cases where MPD called for an ALS unit, only 6% of the calls were potentially life-threatening, 9% were not life-threatening but possibly benefited from ALS, and in 84% of cases, ALS was not needed. Stated another way, MPD said ALS was needed, but in actuality only 16% of cases possibly benefited from ALS care.
Other studies of 11 low-acuity dispatch codes have shown that MPD does a pretty good job of detecting non-emergency cases. However, other than cardiac arrest, MPD did not do well in detecting high-acuity patients.
Even detecting cardiac arrest was problematic. In an Australian study, MPD only identified 76.7% of cardiac arrests. In one study, the MPD cardiac emergencies dispatch code indicated that 104 calls were cardiac-related. Of these, 53.8% were transported by EMS. Of transports, 28.6% were found to have a cardiac problem and 71.4% were found to have a non-cardiac condition. Thus, the positive predictive value of MPD for this protocol was only 28.6%.
A coin flip would give you a 50% predictive value. Interestingly, in another study, using the “obvious death” dispatch code resulted in the code misidentifying 8% of patients. In 5% of these patients, paramedics initiated resuscitation while another 3% had not suffered a cardiac arrest.
Perhaps the most damning study was published in this month’s issue of Academic Emergency Medicine. Canadian researchers evaluated 32 high-acuity dispatch protocols. The study group consisted of 102,582 patients. The overall sensitivity of MPD was 68.2% with a specificity of 66.2%. One half (16 of 32) of all high-acuity protocols performed no better than chance alone at identifying high-acuity patients. Stated another way, flipping a coin is more accurate for determining an EMS response for half of 32 high-acuity MPD protocols.
Now, what about response times? We had all assumed that the sooner an ambulance arrives, the better. However, like other things in EMS practice, this was not quite true. While significant evidence shows that response times less than four minutes are highly associated with improved outcomes, longer response times, especially the common referenced eight-minute response time, are not associated with improved outcome. A recently released study from one of the ambulance regions in the United Kingdom found that improving response times did not improve patient outcomes.
Thus, in my opinion, “high-performance” EMS is based on a deployment system that does no better than a coin flip, uses a priority dispatch system that is only better than a coin flip 50% of the time in high-acuity cases (and then only marginally) to try and meet an eight-minute response time when the evidence shows that only response times less than four minutes are associated with improved outcomes. If this is not a foundation of sand, I don’t know what is.
The typical cry at the “high performance” EMS conference was, “Well, what would you do?” First, that is not for me to answer. It is up to the proponents of SSM and MPD to prove their system works not those who question the practices. Reversing the burden of proof is a common tactic of those who follow pseudoscience.
Like many things in EMS, the practice or device is put on the EMS market without a scientific basis. Then, when empiric science disproves the practice, instead of changing the practice, the proponents question the research or start splitting rhetorical hairs by changing definitions.
The late Jim Page disdained SSM and Public Utility Models (PUMs). But, his mind was open enough that he allowed Jack Stout to publish his writings in JEMS for many years. Once, on a bus between the Orleans Hotel and a steakhouse in Las Vegas, Jim told me that SSM was nothing more than a way of making the most out of a young and expendable workforce and, in the overall scheme of things, it was much ado about nothing. His skepticism was right on. As with many things in EMS, Jim had this all figured out long before the rest of us.
I will agree that SSM improves response times for the most part. And, the difference may be statistically significant. But, more important, it is not clinically-significant. And, unless you can get there in four minutes or less, it really doesn’t matter does it?
Thus, if relatively small increments in response time performance do not yield improvements in clinical outcomes corresponding to costs; and if the impact of achieving those increments is adversely felt in areas like staffing retention, longevity, and the like; and if the same outcome performance can be achieved by less complicated, cumbersome, and costly means; why punish your employees with SSM? You know the answer to this conundrum may be extremely simple. When somebody calls for an ambulance, simply dispatch the closest ambulance. No more, no less.
References
- Stout JL: “System status management: The strategy of ambulance placement.” JEMS. May:22-32, 1983.
- Hough TH: “A view from the street: System status management.” JEMS. 12(12):48-50, 1986.
- Brown LH, Lerner EB, Larmon B, et al: “Accuracy of emergency medical services call volume predictions based on demand pattern analysis (Abstract).” Academic Emergency Medicine. S84, 2006.
- Wilson S, Cooke M, Morrell R, et al: “A systematic review of the evidence supporting the use of priority dispatch of emergency ambulances.” Prehospital Emergency Care. 6:42-49, 2002.
- Kennedy JD, Sweeney TA, Roberts D, et al: “Effectiveness of a medical dispatch protocol for abdominal pain.” Prehospital Emergency Care. 7:89-93, 2003.
- Shah MN, Bishop P, Lerner EB, et al: “Validation of using EMS dispatch codes to identify low-acuity patients.” Prehospital Emergency Care. Jan-Mar;9(1):24-31, 2005.
- Hewerd A, Damiani M, Hartley-Sharpe C: “Does the use of the Advanced Medical Priority Dispatch System affect cardiac arrest detection?” Emergency Medicine Journal. 21:115-118, 2004.
- Flynn J, Archer F, Morgans A: “Sensitivity and specificity of the medical priority dispatch system in detecting cardiac arrest emergency calls in Melbourne.” Prehospital Emergency Care. 21 (suppl 2):72-76, 2006.
- Reily MJ: “Accuracy of a priority medical dispatch system in dispatching cardiac emergencies in a suburban community.” Prehospital Emergency Care. 21 (2 suppl 2):77-81, 2006.
- Harvey L, Woolard M: “Outcome of patients identified as dead (beyond resuscitation) at the point of the emergency call.” Emergency Medicine Journal. 21:367-369, 2004.
- Feldman MJ, Verbeek PR, Lyons DG, et al: “Comparison of the Medical Priority Dispatch System to an out-of-hospital patient acuity score.” Academic Emergency Medicine. 13(9), 2006.
- Price L: “Treating the clock and not the patient: ambulance response times and risk.” Quality and Safety in Health Care. 15:127-130, 2006.
- Pons PT, Haukoos JS, Bloodworth W, et al: “Paramedic response time: does it affect patient survival?” Academic Emergency Medicine. 12:594-600, 2005
- Turner J, O Keefe C, Dixon S, et al: The costs and benefits of changing ambulance response time performance standards. Medical Care Research Unit School of Health and Related Research, University of Sheffield.
