In my previous posts, I shared tips for managing the complexity of branching scenarios and some ideas on how long to let learners go down the wrong path. At some point in that wrong path, you have to redirect learners.
The question is: do you restart the scenario from the beginning or do you go back partway, maybe even just one single step?
Many scenarios I write have at least one short path of wrong decisions. This path usually starts with a poor choice. I give learners the option to correct their mistake and get onto a better path. However, if they make multiple wrong choices without ever making a good one, I force them to restart.
If the whole scenario is short (3-4 decisions on the ideal path), I usually just force a restart after each ending.
Back One Step
Most of the time, especially in longer scenarios, you can allow people to back up in the scenario rather than restarting from the very beginning. If learners make 4 correct decisions in a row but have a mistake in step 5, do they really need to go back to the beginning? Maybe they just need to jump back one step to where they made a mistake.
Cathy Moore uses this approach in many of her scenarios. For example, in this Ethics Training example, you can jump back to the previous decision. She shows this with a “Go Back” link. Sometimes you can only go back after reading feedback (“What happened?”).
Back to a Checkpoint
If your scenario allows people to make multiple wrong choices, you might have “checkpoints” where you return to. Let’s say that Joanna makes 3 correct choices, followed by 3 incorrect ones. If there’s a checkpoint after 3 choices, she can jump back to that point.
Think about video games. If your character dies in level 8, you don’t have to go back and play through levels 1 through 7 again. You either start at level 8 (a checkpoint) or right before you died (back one step).
Mix it up
You can use these techniques together in the same scenario. Some paths might lead to a restart, while minor errors might just return to one step earlier. It’s not an all or nothing question.
Can You Please Help Me?
Can you do me a favor? If you have a question about how to create branching scenarios, can you ask it in the comments? (If you’re reading this in email, feel free to reply and ask privately.) I’ll reply to every question asked. You might also see your question in a future blog post.
In a comment to my post on Managing the Complexity in Branching Scenarios, Nicole Legault made a interesting point. “Why make a learner go so far down a wrong path? I think it’s best to correct and try to get them back on the right (or best) path.”
To some extent, I agree with Nicole. I’m not sure how much value there is to learners in going down seven steps of the wrong path with no way to recover. Where I perhaps disagree is about how the correction should happen. I try to give learners the opportunity to correct their own mistakes. However, that’s different from correcting them and forcing them back on the right path.
There are a couple of ways to handle wrong answers in scenarios.
One way is limited branching. Instead of a true branching scenario with multiple endings, this is essentially a single correct path and a single ending. When you make an incorrect choice, you get some customized feedback and perhaps see limited consequences of your decision. In the long run, there are no real consequences for mistakes. You are forced back to the correct path, regardless of your mistakes.
Although he doesn’t call it that, limited branching is the model explained in Tom Kuhlmann’s Rapid eLearning Blog as an easy way to build scenarios. Tom points out that this model is simpler doesn’t get you overly bogged down in complexity.
In limited branching, you can get the wrong answer every single time, and the scenario still propels you forward. This works OK if your scenario is a series of independent decisions rather than multiple decisions in a single large scenario. If you’re teaching a process with multiple steps, where each step is contingent on the previous step, this method doesn’t create as realistic of an assessment.
Immediate Failure and Restart
The opposite end of the spectrum from limited branching (where you can make endless wrong answers) is immediate, catastrophic failure. If you make a single incorrect decision, you restart the scenario back at the beginning. Personally, I don’t like scenarios where a single wrong answer results in catastrophic failure unless that’s what would happen in real life. Some errors really are major and should result in immediate restarts.
If you’re creating training for nurses, administering 10 times the needed dose of a medication is a catastrophic failure. If you’re creating a scenario to show what to do in an active shooter situation, a decision that results in someone dying is a catastrophic failure. In both of those scenarios, forcing learners to restart at the beginning is appropriate.
2 or 3 Consecutive Wrong Answers
Most of the time in scenarios, we’re working with gray area. In real life, we often have opportunities to change paths and correct mistakes. Where a single isolated mistake can be corrected, the cumulative effect of several wrong answers is the real concern.
In my scenarios, I usually try to limit it to two or three consecutive wrong answers before a restart. I give people opportunities to get back on the right path by making better choices. If they keep going down the wrong path, they have to restart and try again. I won’t force them to correct; learners need the opportunity to fail.
In this example, there are good (green), OK (orange), and bad (red) choices. If you choose C (red) at the beginning, you may reach a poor ending after just 2 choices. However, if you improve your choices, you can get back to a good (green) choice by correcting your mistakes.
Limiting it to two or three consecutive wrong answers also helps limit the complexity of branching scenarios. You don’t have to create a full-length path of increasingly wrong answers.
Giving people a short, but incorrect (or partially incorrect), path also gives you the opportunity to show delayed consequences.
What do you do?
How do you handle wrong answers in a branching scenario? How long do you let learners go down an incorrect path before either forcing a restart or forcing them back on the correct path?
The traditional multiple choice questions we use in assessment are often abstract and measure only whether people recall facts they heard in the last 5 minutes. Converting these questions to scenario-based questions can increase the level of difficulty, measure higher level thought, and provide relevant context.
Let’s say you’re creating training for managers on how to provide reasonable accommodations for employees. You drafted a set of traditional multiple choice questions as a quiz for the end of the course, but they’re all very low level. You want to improve the quality of your assessment with some scenarios.
This is a question from your current quiz that measures recall of a fact from the training. The rest of the assessment is similar.
Example 1 (Original)
What reasonable accommodation is recommended for a temporary disability or medical issue affecting work?
- None; reasonable accommodations are only used for permanent or long-term disabilities.
- Unpaid time off can be offered as an accommodation for temporary issues.
- Paid time off should be offered, even if it exceeds the amount of paid time off other employees receive.
Align to Objectives
What are your objectives? Does your assessment align to them? If not, rewrite it.
In this example, the objective is “The learner will follow the procedure for providing reasonable accommodations.” The objective is application level; you need to apply this procedure. (You could argue for analysis or evaluation here too, but let’s assume it’s application.)
The question assesses recall; the objective requires application. Therefore, this question should be rewritten at a higher level.
When would people use this?
The first step to shifting from traditional to scenario-based assessment is asking when people would use the information. When would managers need to know about handling temporary disabilities? A common situation would be due to an illness or surgery. Maybe an employee needs a reduced schedule due to fatigue from chemo. Maybe an employee needs time off to recover from back surgery.
For each multiple choice question, ask yourself how learners would use that information on the job. When would they need to differentiate between those options?
If you can’t come up with any situation in which people would need this information on the job, why are you asking that question? If you have a question with just irrelevant information, skip down to the section on complete rewrites below.
Scenario as Introduction
One method to revise the question is to add a scenario to introduce the choices. This provides context. It shifts the question from just recalling information to using that information to make a decision.
Let’s see how this works with the previous example. The scenario introduces the question. The choices are essentially the same as before, but now it’s a decision about how to work with an employee you manage. Instead of measuring recall, this question measures if learners can apply the reasonable accommodations policy.
Example 1 (Revised)
Simon, a graphic designer on the team you manage, is having surgery. He requested 2 weeks time off to recover after his surgery. How should you respond?
- Let Simon know he can use his accrued vacation time. Reasonable accommodations are only used for permanent or long-term disabilities.
- Provide two weeks unpaid time off.
- Provide two weeks paid time off.
Notice that this scenario isn’t long; it’s only 2 more sentences than the original question.
Sometimes adding a scenario at the beginning won’t work, and you need a complete rewrite of the question. If the question is something unrelated to your objectives or that people will never use on the job, you have to start over and replace the question.
Look at this example. Would a manager ever need to know this history on the job? Will they be more effective at offering accommodations if they can memorize this date?
Example 2 (Original)
In what year was the Americans with Disabilities Act or ADA passed by Congress?
We have all seen questions like this on quizzes before. They’re easy to write, but they don’t assess anything meaningful. Replacing it with a scenario-based question would give you a more accurate assessment.
Example 2 (Replacement)
One of your employees, Miranda, brought documentation from her ophthalmologist about her vision and how it affects her driving. Her night vision is deteriorating. Miranda has requested a change in her work schedule. She wants to start and end her work day later to avoid driving in the early morning when it’s still dark. How do you respond?
- Agree to adjust Miranda’s schedule.
- Tell Miranda to contact HR to start the official accommodation process.
- Tell Miranda that the schedule change is not possible since it creates too much burden on the rest of the team.
What Do You Want to Learn?
What else would you like to learn about writing these kinds of assessment questions? Do you have questions I could answer in a future post? Let me know in the comments.