Without a mechanism to evaluate your students, you’ll never know if you’ve achieved the goals and objectives of instruction. So EMS educators must be familiar with the concepts of evaluation and how to construct valid and reliable exams. In its simplest form, a test is a written or situational occurrence that allows the learner to demonstrate learning. A test is also helpful because it provides feedback to the instructor and student, clarifying and redirecting future learning needs, and evaluating mastery of the content at a minimally acceptable level of competence by comparing student performance to pre-established criteria.
An exam that is well written has carefully crafted questions that accomplish several goals:
- Items demonstrate essential knowledge (high-frequency/high-criticality) .
- Items reflect expert consensus on best or correct answers.
- Examinee task is complete and specifically defined.
- The exam includes many questions that can be answered briefly.
- The instructor has framed the ideal answers in the key.
- All students answer the same questions or at least answer similar questions that measure the same objectives.
You can use a wide variety of question types: multiple choice, true/false, matching, short answer completion, multiple choice, essay and terminal/certifying. Several terms are used to identify the components of the question and subsequent answers.
One common term is the stem, also referred to as the question. The stem formulates a problem or asks a question. It may be written as a question or an incomplete statement. Within the set of responses are the distracters, options and key. Distracters are a series of alternative options that provide plausible but incorrect or less desirable responses to the stem. The term option is used as a collective for the distracters and the key. The key is the correct or best response. The last term is an item. An item is a common instructional design term for all the components of a written examination question, including the stem, key and distracters.
When you’re constructing an exam, there are some general guidelines to follow. To begin, you should create a blueprint. A valid test must measure the achievement of specific knowledge objectives and test what it’s intended to test. To do this, it should be developed from a blueprint. A blueprint is a well-designed plan to measure learning.
First, determine the number of questions to include. This may be driven by the amount of content that has been covered, time limitations of the course schedule, the nature of the quiz/exam or policy.
Final course exams generally have 150–200 questions. Beyond that, students become fatigued and make unnecessary errors. Continued education (CE) post-tests and daily or weekly class quizzes are usually variable in number. Exams usually allow one minute per question unless using complex scenarios, so use that rule of thumb when creating testing instruments.
Next, identify instructional objectives the test should measure. These should be based on a needs-assessment that may include quality improvement (QI) data, a practice analysis of highly critical concepts or the important class objectives to be measured based on the educator’s expert opinion.
Structure the exam in a logical manner. Design a plan that orders the collective behaviors and content areas to be included in the test. Group items of a similar content area together. Have sections follow a linear or logical sequence. For a final exam, list the major modules or subsections of the curriculum and place them as subtitles on the left side of the blueprint.
Then determine the relative importance (value) of each behavior or content area. This may be determined by content experts, a practice analysis, the number of hours devoted to each subject in the class and the criticality to practice. For example, airway questions may outnumber radio communication questions on a paramedic class modular exam.
Also, if you’re creating an exam with a limited number of questions (e.g., final exam or CE post-test), only include critical concepts and don’t waste time on lower level questions.
It’s important to predetermine the percentages of each question type. For example, a balanced exam may have approximately 20% of the items measure the learner’s knowledge relative to conducting a patient assessment, 20% measure interpretation of data to reach an appropriate conclusion or impression, 40% evaluate patient interventions, 15% measure facts and 5% measure ongoing assessments or patient responses. This ensures that the exam tests across a provider’s major scopes of practice. You don’t want to include too many easy (factual only) questions.
Assessment questions evaluate an examinee’s ability to collect data regarding a patient’s actual or potential health needs. Items may focus on the following behaviors:
- Primary and secondary assessments;
- Obtaining a history (e.g, SAMPLE);
- Techniques and elements of the physical examination (e.g., inspection, palpation, percussion and auscultation);
- Diagnostic studies within the examinee’s scope of practice (e.g., ECG, pulse oximetry, capnography and glucose monitoring).
These questions evaluate a student’s ability to ask the right questions, gather the right information and perform a physical exam appropriately. Here’s a sample assessment question:
What should be an EMT’s first action when arriving at a call?
A. obtain a SAMPLE history
B. perform a detailed assessment
C. take a full set of vital signs
D. conduct a scene size-up
Analysis/interpretation of data questions measure an examinee’s ability to form an appropriate conclusion by organizing, analyzing, synthesizing and/or summarizing the data collected. These questions test the person’s ability to interpret assessment data, including clinical findings, vital sign values and diagnostic studies.
Here’s an example:
A conscious adult with blunt chest trauma presents with extreme dyspnea, agitation and restlessness. Breath sounds are absent on the right, diminished on the left, and neck veins are distended. Carotid pulse is weak and rapid; radial pulse is absent. Heart tones are clear, and there’s no paradoxical movement of the chest wall. What should an EMT suspect?
A. flail chest
B. cardiac tamponade
C. open pneumothorax
D. tension pneumothorax
Intervention questions should test the student’s ability to appropriately treat the patient or select a course of action. Interventions should be patient-focused and goal-directed based on physiologic principles. On written exams, these items test the cognitive aspects of interventions to be performed or anticipated (e.g., airway management, suction and CPR).
In which is a traction splint indicated?
A. Deformity of the knee with displacement of the patella
B. Shortened and externally rotated leg with severe hip pain
C. Open fracture with bone protruding from the anterior tibia
D. Closed deformity of the middle thigh with decreased pedal pulse
Evaluation questions assess a patient’s responses to interventions, which could include side effects of medications, responses to fluid boluses, effectiveness of pain management, or airway status after adjuncts are applied. These questions determine if the examinee recognizes patient responses to an intervention that are anticipated or indicate that they need further monitoring or interventions.
If breath sounds are present on the right and absent on the left after tracheal intubation, what action is indicated?
A. withdraw the tube slightly and re-auscultate breath sounds
B. needle decompress the left lung to relieve excess air tension
C. deflate the cuff, remove the tube and begin the process again
D. aspirate the tube with an esophageal detector device to check for resistance
Once you’ve addressed the organization of the exam and the types of questions to use, the next step is to carefully consider the exam’s content. Here are some helpful tips:
- Paraphrase language used during instruction in the test items to avoid testing for simple recall, but don’t quote directly from a book or outline.
- Keep the content of each item independent from content of other items on the test.
- Ensure a correct answer on one item doesn’t divulge the answer for another.
- Avoid over-specific and over-general content with multiple choice items.
- Steer clear of opinion-based items.
- Avoid trick questions.
- Use simple vocabulary that’s appropriate for the group being tested (no higher than 12th grade reading level for EMT and paramedic students).
- Avoid use of acronyms or abbreviations unless they’re universally understood.
Ensure the item can be answered without looking at the options, that each option is 100% correct or incorrect or that it’s listed in a data dictionary for the exam.
Another factor to consider is style. Be consistent in the design strategies used for margins, question formatting and graphics (fonts). Clean, crisp fonts duplicate best and are easiest to read (e.g., Arial font 10 or 11). Instructors should be consistent with the use of underlining, italics, boldface fonts or capitalization for emphasis. Instructors should also vertically format items rather than horizontally because it’s easier to read.
Another tip is to provide clear, complete directions on the cover. Indicate whether a student can write on the test booklet or answer sheet. Also indicate on the cover sheet whether any electronic equipment or tools (calipers) are allowed. State if the students must use a pencil to fill in an answer sheet.
You can also indicate whether breaks are allowed during the test. Be sure to specify consequences if the instructor verifies academic dishonesty, if an exam is compromised or removed from the testing environment.
Even the most seasoned educators can cringe when it comes to measuring learning by creating valid and reliable written quizzes and exams. These suggestions form the tip of the iceberg with respect to item writing and exam construction but offer a sound foundation from which to begin. In reality, it just takes adherence to some simple guidelines and attention to detail to take your tests from failing to fabulous. JEMS
This article originally appeared in June 2010 JEMS as “All Access Pass: Mastering the use of IO devices.”