Category: Storytelling & Scenarios

Immediate and Delayed Consequences in Branching Scenarios

In branching scenarios, we can use a combination of immediate and delayed consequences and feedback. Consequences are what happens as a result of decisions; feedback is what we tell learners after decisions.

Immediate & Delayed Consequences

Use Immediate Consequences Often

Immediate consequences are the intrinsic effects of decisions. A customer who responds angrily, software that doesn’t produce the desired result, or time lost on a project could all be immediate consequences. These consequences don’t directly tell the learner, “Sorry, that was incorrect.” Learners have to perceive and understand the cues in the scenario. They have to draw conclusions based on those cues.

If your learners will need to follow cues when they apply what they’re learning, it’s helpful to provide real-world consequences in your scenario. It’s beneficial to practice interpreting cues.

Immediate consequences that simulate real-world cues can also be more engaging than the omniscient narrator dictating what you did right or wrong.  It keeps learners in the mindset of the story without hitting them over the head with a reminder that they’re learning something.

Use Immediate Feedback with Novices

Immediate feedback is different from intrinsic consequences. This is the instructional feedback or coaching that directly tells learners why their decisions are right or wrong. While this can pull people out of the “flow” of a story, immediate feedback can be helpful in some situations.

First, novice learners who are still building mental models of a topic may benefit more from immediate feedback. Novices may not have the expertise to sort through real-world cues and draw accurate conclusions from them.  Therefore, it may be more important to provide immediate feedback after each decisions in a branching scenario if your audience is new to the topic.

In his research report “Providing Learners with Feedback,” Will Thalheimer explains the benefits of immediate feedback for novices.

“On the surface of it, it just doesn’t make sense that when a learner is piecing together arrays of building blocks into a fully-formed complex concept, they wouldn’t need some sort of feedback as they build up from prerequisite concepts. If the conceptual foundation they build for themselves is wrong, adding to that faulty foundation is problematic. Feedback provided before these prerequisite mental modelettes are built should keep learners from flailing around too much. For this reason, I will tentatively recommend immediate feedback as learners build understanding.”

Provide Instructional Feedback Before a Retry

I always use feedback before restarting a scenario.  If a learner has reached an unsatisfactory ending in a scenario, it’s beneficial to do a short debrief of their decisions and what went wrong. Especially for more experienced learners, some of that feedback may be delayed from when they made the decision. You can summarize the feedback for several previous decisions on the path that led to the final decision.

This feedback should happen before they are faced with the same scenario decisions again. Otherwise, they could make the same mistakes again (reinforcing those mistakes) or simply guess without gaining understanding.

Thalheimer’s research also supports this.

“When learners get an answer wrong or practice a skill inappropriately, we ought to give them feedback before they attempt to re-answer the question or re-attempt the skill. This doesn’t necessarily mean that we should give them immediate feedback, but it does mean that we don’t want to delay feedback until after they are faced with additional retrieval opportunities.”

Use Delayed Feedback with Experienced Learners

Thalheimer notes that delayed feedback may be more effective for retention (i.e., how much do learners remember). That effect might be due to the spacing effect (that is, reviewing content multiple times, spaced out over times, is better for learning than cramming everything into a single event). The delay for feedback doesn’t have to be long; one study mentioned in Thalheimer’s report showed that delaying feedback by 10 seconds improved outcomes.

Delayed feedback may also be more appropriate for experienced learners who are improving existing skills rather than novices building new skills. Experienced learners already have mental models in place, so they don’t have the same needs for immediate correction as novices. They can get the benefit of delayed feedback.

Use Delayed Feedback with Immediate Consequences

In branching scenarios, we can use a combination of immediate intrinsic consequences (e.g., an angry customer response) and delayed instructional feedback (e.g., you didn’t acknowledge the customer’s feelings). Feedback before a retry or restart could count as delayed if it includes feedback for multiple decisions. If you let learners make 2 or 3 wrong choices before a restart, the combined feedback will effectively be delayed.

Use Delayed Consequences When Realistic

We  don’t always immediately know our mistakes are wrong in real life. Sometimes the consequence isn’t obvious right away. Sometimes it seems like a gain the short run, but causes problems in the long run. If that’s the kind of situation you’re training for, letting people continue on the wrong path for a little while makes sense. Neither limited branching nor immediate failure allow you to show delayed consequences.

Providing these delayed consequences has the advantage of better learning from delayed feedback, plus it creates a more realistic and engaging story. Delayed consequences shouldn’t be forced into a scenario where it’s not realistic, but they are a good way to show the long-term effects of actions.

Think about how delayed consequences could be shown in these examples:

  • A bartender gives away many free drinks. The immediate consequence is that the customers are happy, but the delayed consequence is a loss of profit for the bar.
  • A sales associate sells a customer a product that is less expensive but meets the customer’s needs. The immediate consequence is that the sales associate makes less commission that day, but the delayed consequence is that the customer is loyal and refers 2 friends. In this case, the total commission earned is higher even though the immediate sale was lower.
  • A doctor could skip a screening question with a patient. The immediate consequence is finding something that looks like the problem, but the delayed consequence is the actual underlying problem remaining.
  • A manager asks an ID to create training. The ID gets started building it right away, trusting that the team requesting the training knows their needs. The immediate consequence is a happy manager, but the delayed consequence is ineffective training that doesn’t actually solve the business problem.
  • If you’re teaching ethics, a small ethical lapse early in the scenario might not seem like a big deal. The immediate consequence might be meeting a deadline or increased recognition.  In the long run, that small lapse leads a continued need to cover up your actions. The Lab: Avoiding Research Misconduct is an example with delayed consequences in some paths.

Looking for More?

Read more about branching scenarios:

 

Don’t Restart Scenario-Based Learning, Go Back

In my previous posts, I shared tips for managing the complexity of branching scenarios and some ideas on how long to let learners go down the wrong path. At some point in that wrong path, you have to redirect learners.

The question is: do you restart the scenario from the beginning or do you go back partway, maybe even just one single step?

Restart or Go Back?

Restart

Many scenarios I write have at least one short path of wrong decisions. This path usually starts with a poor choice. I give learners the option to correct their mistake and get onto a better path. However, if they make multiple wrong choices without ever making a good one, I force them to restart.

If the whole scenario is short (3-4 decisions on the ideal path), I usually just force a restart after each ending.

Back One Step

Most of the time, especially in longer scenarios, you can allow people to back up in the scenario rather than restarting from the very beginning. If learners make 4 correct decisions in a row but have a mistake in step 5, do they really need to go back to the beginning? Maybe they just need to jump back one step to where they made a mistake.

Cathy Moore uses this approach in many of her scenarios. For example, in this Ethics Training example, you can jump back to the previous decision. She shows this with a “Go Back” link. Sometimes you can only go back after reading feedback (“What happened?”).

Back to a Checkpoint

If your scenario allows people to make multiple wrong choices, you might have “checkpoints” where you return to. Let’s say that Joanna makes 3 correct choices, followed by 3 incorrect ones. If there’s a checkpoint after 3 choices, she can jump back to that point.

Think about video games. If your character dies in level 8, you don’t have to go back and play through levels 1 through 7 again. You either start at level 8 (a checkpoint) or right before you died (back one step).

Mix it up

You can use these techniques together in the same scenario. Some paths might lead to a restart, while minor errors might just return to one step earlier. It’s not an all or nothing question.

Can You Please Help Me?

Can you do me a favor? If you have a question about how to create branching scenarios, can you ask it in the comments? (If you’re reading this in email, feel free to reply and ask privately.) I’ll reply to every question asked. You might also see your question in a future blog post.

How Long Should We Let Learners Go Down the Wrong Path?

In a comment to my post on Managing the Complexity in Branching Scenarios, Nicole Legault made a interesting point. “Why make a learner go so far down a wrong path? I think it’s best to correct and try to get them back on the right (or best) path.”

To some extent, I agree with Nicole. I’m not sure how much value there is to learners in going down seven steps of the wrong path with no way to recover. Where I perhaps disagree is about how the correction should happen. I try to give learners the opportunity to correct their own mistakes. However, that’s different from correcting them and forcing them back on the right path.

There are a couple of ways to handle wrong answers in scenarios.

Limited Branching

One way is limited branching. Instead of a true branching scenario with multiple endings, this is essentially a single correct path and a single ending. When you make an incorrect choice, you get some customized feedback and perhaps see limited consequences of your decision. In the long run, there are no real consequences for mistakes. You are forced back to the correct path, regardless of your mistakes.

Although he doesn’t call it that, limited branching is the model explained in Tom Kuhlmann’s Rapid eLearning Blog as an easy way to build scenarios. Tom points out that this model is simpler doesn’t get you overly bogged down in complexity.

In limited branching, you can get the wrong answer every single time, and the scenario still propels you forward. This works OK if your scenario is a series of independent decisions rather than multiple decisions in a single large scenario. If you’re teaching a process with multiple steps, where each step is contingent on the previous step, this method doesn’t create as realistic of an assessment.

Immediate Failure and Restart

The opposite end of the spectrum from limited branching (where you can make endless wrong answers) is immediate, catastrophic failure. If you make a single incorrect decision, you restart the scenario back at the beginning. Personally, I don’t like scenarios where a single wrong answer results in catastrophic failure unless that’s what would happen in real life. Some errors really are major and should result in immediate restarts.

If you’re creating training for nurses, administering 10 times the needed dose of a medication is a catastrophic failure. If you’re creating a scenario to show what to do in an active shooter situation, a decision that results in someone dying is a catastrophic failure. In both of those scenarios, forcing learners to restart at the beginning is appropriate.

 

2 or 3 Consecutive Wrong Answers

Most of the time in scenarios, we’re working with gray area. In real life, we often have opportunities to change paths and correct mistakes. Where a single isolated mistake can be corrected, the cumulative effect of several wrong answers is the real concern.

In my scenarios, I usually try to limit it to two or three consecutive wrong answers before a restart. I give people opportunities to get back on the right path by making better choices. If they keep going down the wrong path, they have to restart and try again. I won’t force them to correct; learners need the opportunity to fail.

In this example, there are good (green), OK (orange), and bad (red) choices. If you choose C (red) at the beginning, you may reach a poor ending after just 2 choices. However, if you improve your choices, you can get back to a good (green) choice by correcting your mistakes.

Flow chart showing that two consecutive wrong answers lead to a poor ending Flowchart showing a branching scenario with an opportunity to recover from mistakes

Limiting it to two or three consecutive wrong answers also helps limit the complexity of branching scenarios. You don’t have to create a full-length path of increasingly wrong answers.

Giving people a short, but incorrect (or partially incorrect), path also gives you the opportunity to show delayed consequences.

 

 

 

What do you do?

How do you handle wrong answers in a branching scenario? How long do you let learners go down an incorrect path before either forcing a restart or forcing them back on the correct path?

Converting Traditional Multiple Choice Questions to Scenario-Based Questions

The traditional multiple choice questions we use in assessment are often abstract and measure only whether people recall facts they heard in the last 5 minutes. Converting these questions to scenario-based questions can increase the level of difficulty, measure higher level thought, and provide relevant context.

Converting to Scenario-Based Questions

Example Question

Let’s say you’re creating training for managers on how to provide reasonable accommodations for employees. You drafted a set of traditional multiple choice questions as a quiz for the end of the course, but they’re all very low level. You want to improve the quality of your assessment with some scenarios.

This is a question from your current quiz that measures recall of a fact from the training. The rest of the assessment is similar.

Example 1 (Original)

What reasonable accommodation is recommended for a temporary disability or medical issue affecting work?

  1. None; reasonable accommodations are only used for permanent or long-term disabilities.
  2. Unpaid time off can be offered as an accommodation for temporary issues.
  3. Paid time off should be offered, even if it exceeds the amount of paid time off other employees receive.

Align to Objectives

What are your objectives? Does your assessment align to them? If not, rewrite it.

In this example, the objective is “The learner will follow the procedure for providing reasonable accommodations.” The objective is application level; you need to apply this procedure. (You could argue for analysis or evaluation here too, but let’s assume it’s application.)

The question assesses recall; the objective requires application. Therefore, this question should be rewritten at a higher level.

When would people use this?

The first step to shifting from traditional to scenario-based assessment is asking when people would use the information. When would managers need to know about handling temporary disabilities? A common situation would be due to an illness or surgery. Maybe an employee needs a reduced schedule due to fatigue from chemo. Maybe an employee needs time off to recover from back surgery.

For each multiple choice question, ask yourself how learners would use that information on the job. When would they need to differentiate between those options?

If you can’t come up with any situation in which people would need this information on the job, why are you asking that question? If you have a question with just irrelevant information, skip down to the section on complete rewrites below.

Scenario as Introduction

One method to revise the question is to add a scenario to introduce the choices. This provides context. It shifts the question from just recalling information to using that information to make a decision.

Let’s see how this works with the previous example. The scenario introduces the question. The choices are essentially the same as before, but now it’s a decision about how to work with an employee you manage. Instead of measuring recall, this question measures if learners can apply the reasonable accommodations policy.

Example 1 (Revised)

Simon, a graphic designer on the team you manage, is having surgery. He requested 2 weeks time off to recover after his surgery. How should you respond?

  1. Let Simon know he can use his accrued vacation time. Reasonable accommodations are only used for permanent or long-term disabilities.
  2. Provide two weeks unpaid time off.
  3. Provide two weeks paid time off.

Notice that this scenario isn’t long; it’s only 2 more sentences than the original question.

Complete Replacement

Sometimes adding a scenario at the beginning won’t work, and you need a complete rewrite of the question. If the question is something unrelated to your objectives or that people will never use on the job, you have to start over and replace the question.

Look at this example. Would a manager ever need to know this history on the job? Will they be more effective at offering accommodations if they can memorize this date?

Example 2 (Original)

In what year was the Americans with Disabilities Act or ADA passed by Congress?

  1. 1985
  2. 1990
  3. 1995
  4. 2000

We have all seen questions like this on quizzes before. They’re easy to write, but they don’t assess anything meaningful. Replacing it with a scenario-based question would give you a more accurate assessment.

Example 2 (Replacement)

One of your employees, Miranda, brought documentation from her ophthalmologist about her vision and how it affects her driving. Her night vision is deteriorating. Miranda has requested a change in her work schedule. She wants to start and end her work day later to avoid driving in the early morning when it’s still dark. How do you respond?

  1. Agree to adjust Miranda’s schedule.
  2. Tell Miranda to contact HR to start the official accommodation process.
  3. Tell Miranda that the schedule change is not possible since it creates too much burden on the rest of the team.

What Do You Want to Learn?

What else would you like to learn about writing these kinds of assessment questions? Do you have questions I could answer in a future post? Let me know in the comments.

Managing the Complexity of Branching Scenarios

On reddit, someone asked how to manage the complexity of branching scenarios and keep them from growing out of control. One of the issues with branching scenarios is that you can get exponential growth. If each choice has 3 options, you end up with 9 slides after just 2 choices, and 27 after 3 choices. This is 40 pages total with only 3 decisions per path. For most projects, that’s more complexity than you want or need.

Branching scenario with exponential growth

So how do you manage this complexity?

Use Twine

One way to make this branching easier to manage is by creating your scenarios in Twine. Twine makes it very easy to draft scenarios and check how all the connections flow together. No matter how complex your scenario is, Twine makes it easier to create it. Cathy Moore has an example of a scenario she built in Twine. This scenario has 57 total decision points, but it only took her 8 hours to create.

You can use Twine as your initial prototype, or you can use it as your final product. I have used Twine as my initial draft and prototype, then exported everything to Word as a storyboard for developers to build the final version in Storyline.

Planning a Scenario

Before I sit down to write a scenario, I always know my objectives. What are you teaching or assessing?

I usually have an idea of how long the ideal or perfect path will be. If you have a multi-step process, that’s your ideal path. If there’s going to be 4 decision points on the shortest path, I know what those are before I start writing.

I also usually know at least some of the decision points based on errors or mistakes I need to address.

There’s a limit to how much you can plan before you just start writing it out though. I find it’s easier to just open up Twine and figure it out within that system.

Allow Opportunities to Fix Mistakes

One trick for managing the potentially exponential growth is by giving learners a chance to get back on the right path if they make a minor error. If they make 2 or 3 errors in a row, they get to an ending and have to restart the whole thing.

For example, maybe you’re teaching a communication skill where they should start with an open-ended question before launching into a sales pitch. Choice A is the open-ended question (the best choice). Choice B is a closed question (an OK choice). Choice C is jumping right into the sales pitch without asking (bad choice). After the customer response for choice B, I’d give them an opportunity to use the open-ended question (A) as their follow up. Reusing some choices helps keep it from growing out of control. In this image, reusing choices cuts the total number of pages from 40 to 20.

Flowchart for branching scenario with 20 pages Make Some Paths Shorter

Not every path needs to be the same length. In the above image, one branch from choice C is shorter. It ends after 2 choices instead of 3. You might make a short path if people make several major errors in a row. Past a certain point, it makes sense to  ask people to reset the scenario from the beginning or backtrack to a previous decision.

Good, OK, and Bad

In branching scenarios, not everything is as black and white as a clear-cut right or wrong answer. You can have good, OK, and bad choices and endings. In this example from my portfolio, green is good choices/endings, orange is OK choices/endings, and red is bad choices/endings. In this scenario, if you choose 39 (bad), you have 3 options: 40 (back on the good path, recovering from the mistake), 41 (OK), and 42 (a bad choice leading to a restart). This example has 15 endings, which is still more than I would like; if I was redoing it now I would probably collapse a few more of those endings together.

Branching scenario flowchart with good, OK, and bad choices and endings

Your Ideas?

Do you have any suggestions or tips for managing and reducing the complexity of branching scenarios? Please share in the comments.