How Long Should We Let Learners Go Down the Wrong Path?

In a comment to my post on Managing the Complexity in Branching Scenarios, Nicole Legault made a interesting point. “Why make a learner go so far down a wrong path? I think it’s best to correct and try to get them back on the right (or best) path.”

To some extent, I agree with Nicole. I’m not sure how much value there is to learners in going down seven steps of the wrong path with no way to recover. Where I perhaps disagree is about how the correction should happen. I try to give learners the opportunity to correct their own mistakes. However, that’s different from correcting them and forcing them back on the right path.

There are a couple of ways to handle wrong answers in scenarios.

Limited Branching

One way is limited branching. Instead of a true branching scenario with multiple endings, this is essentially a single correct path and a single ending. When you make an incorrect choice, you get some customized feedback and perhaps see limited consequences of your decision. In the long run, there are no real consequences for mistakes. You are forced back to the correct path, regardless of your mistakes.

Although he doesn’t call it that, limited branching is the model explained in Tom Kuhlmann’s Rapid eLearning Blog as an easy way to build scenarios. Tom points out that this model is simpler doesn’t get you overly bogged down in complexity.

In limited branching, you can get the wrong answer every single time, and the scenario still propels you forward. This works OK if your scenario is a series of independent decisions rather than multiple decisions in a single large scenario. If you’re teaching a process with multiple steps, where each step is contingent on the previous step, this method doesn’t create as realistic of an assessment.

Immediate Failure and Restart

The opposite end of the spectrum from limited branching (where you can make endless wrong answers) is immediate, catastrophic failure. If you make a single incorrect decision, you restart the scenario back at the beginning. Personally, I don’t like scenarios where a single wrong answer results in catastrophic failure unless that’s what would happen in real life. Some errors really are major and should result in immediate restarts.

If you’re creating training for nurses, administering 10 times the needed dose of a medication is a catastrophic failure. If you’re creating a scenario to show what to do in an active shooter situation, a decision that results in someone dying is a catastrophic failure. In both of those scenarios, forcing learners to restart at the beginning is appropriate.

 

2 or 3 Consecutive Wrong Answers

Most of the time in scenarios, we’re working with gray area. In real life, we often have opportunities to change paths and correct mistakes. Where a single isolated mistake can be corrected, the cumulative effect of several wrong answers is the real concern.

In my scenarios, I usually try to limit it to two or three consecutive wrong answers before a restart. I give people opportunities to get back on the right path by making better choices. If they keep going down the wrong path, they have to restart and try again. I won’t force them to correct; learners need the opportunity to fail.

In this example, there are good (green), OK (orange), and bad (red) choices. If you choose C (red) at the beginning, you may reach a poor ending after just 2 choices. However, if you improve your choices, you can get back to a good (green) choice by correcting your mistakes.

Flow chart showing that two consecutive wrong answers lead to a poor ending Flowchart showing a branching scenario with an opportunity to recover from mistakes

Limiting it to two or three consecutive wrong answers also helps limit the complexity of branching scenarios. You don’t have to create a full-length path of increasingly wrong answers.

Giving people a short, but incorrect (or partially incorrect), path also gives you the opportunity to show delayed consequences.

 

 

 

What do you do?

How do you handle wrong answers in a branching scenario? How long do you let learners go down an incorrect path before either forcing a restart or forcing them back on the correct path?

Converting Traditional Multiple Choice Questions to Scenario-Based Questions

The traditional multiple choice questions we use in assessment are often abstract and measure only whether people recall facts they heard in the last 5 minutes. Converting these questions to scenario-based questions can increase the level of difficulty, measure higher level thought, and provide relevant context.

Converting to Scenario-Based Questions

Example Question

Let’s say you’re creating training for managers on how to provide reasonable accommodations for employees. You drafted a set of traditional multiple choice questions as a quiz for the end of the course, but they’re all very low level. You want to improve the quality of your assessment with some scenarios.

This is a question from your current quiz that measures recall of a fact from the training. The rest of the assessment is similar.

Example 1 (Original)

What reasonable accommodation is recommended for a temporary disability or medical issue affecting work?

  1. None; reasonable accommodations are only used for permanent or long-term disabilities.
  2. Unpaid time off can be offered as an accommodation for temporary issues.
  3. Paid time off should be offered, even if it exceeds the amount of paid time off other employees receive.

Align to Objectives

What are your objectives? Does your assessment align to them? If not, rewrite it.

In this example, the objective is “The learner will follow the procedure for providing reasonable accommodations.” The objective is application level; you need to apply this procedure. (You could argue for analysis or evaluation here too, but let’s assume it’s application.)

The question assesses recall; the objective requires application. Therefore, this question should be rewritten at a higher level.

When would people use this?

The first step to shifting from traditional to scenario-based assessment is asking when people would use the information. When would managers need to know about handling temporary disabilities? A common situation would be due to an illness or surgery. Maybe an employee needs a reduced schedule due to fatigue from chemo. Maybe an employee needs time off to recover from back surgery.

For each multiple choice question, ask yourself how learners would use that information on the job. When would they need to differentiate between those options?

If you can’t come up with any situation in which people would need this information on the job, why are you asking that question? If you have a question with just irrelevant information, skip down to the section on complete rewrites below.

Scenario as Introduction

One method to revise the question is to add a scenario to introduce the choices. This provides context. It shifts the question from just recalling information to using that information to make a decision.

Let’s see how this works with the previous example. The scenario introduces the question. The choices are essentially the same as before, but now it’s a decision about how to work with an employee you manage. Instead of measuring recall, this question measures if learners can apply the reasonable accommodations policy.

Example 1 (Revised)

Simon, a graphic designer on the team you manage, is having surgery. He requested 2 weeks time off to recover after his surgery. How should you respond?

  1. Let Simon know he can use his accrued vacation time. Reasonable accommodations are only used for permanent or long-term disabilities.
  2. Provide two weeks unpaid time off.
  3. Provide two weeks paid time off.

Notice that this scenario isn’t long; it’s only 2 more sentences than the original question.

Complete Replacement

Sometimes adding a scenario at the beginning won’t work, and you need a complete rewrite of the question. If the question is something unrelated to your objectives or that people will never use on the job, you have to start over and replace the question.

Look at this example. Would a manager ever need to know this history on the job? Will they be more effective at offering accommodations if they can memorize this date?

Example 2 (Original)

In what year was the Americans with Disabilities Act or ADA passed by Congress?

  1. 1985
  2. 1990
  3. 1995
  4. 2000

We have all seen questions like this on quizzes before. They’re easy to write, but they don’t assess anything meaningful. Replacing it with a scenario-based question would give you a more accurate assessment.

Example 2 (Replacement)

One of your employees, Miranda, brought documentation from her ophthalmologist about her vision and how it affects her driving. Her night vision is deteriorating. Miranda has requested a change in her work schedule. She wants to start and end her work day later to avoid driving in the early morning when it’s still dark. How do you respond?

  1. Agree to adjust Miranda’s schedule.
  2. Tell Miranda to contact HR to start the official accommodation process.
  3. Tell Miranda that the schedule change is not possible since it creates too much burden on the rest of the team.

What Do You Want to Learn?

What else would you like to learn about writing these kinds of assessment questions? Do you have questions I could answer in a future post? Let me know in the comments.

ID and eLearning Links (7/23/17)

Posted from Diigo. The rest of my favorite links are here.

Instructional Design and E-Learning Links

Managing the Complexity of Branching Scenarios

On reddit, someone asked how to manage the complexity of branching scenarios and keep them from growing out of control. One of the issues with branching scenarios is that you can get exponential growth. If each choice has 3 options, you end up with 9 slides after just 2 choices, and 27 after 3 choices. This is 40 pages total with only 3 decisions per path. For most projects, that’s more complexity than you want or need.

Branching scenario with exponential growth

So how do you manage this complexity?

Use Twine

One way to make this branching easier to manage is by creating your scenarios in Twine. Twine makes it very easy to draft scenarios and check how all the connections flow together. No matter how complex your scenario is, Twine makes it easier to create it. Cathy Moore has an example of a scenario she built in Twine. This scenario has 57 total decision points, but it only took her 8 hours to create.

You can use Twine as your initial prototype, or you can use it as your final product. I have used Twine as my initial draft and prototype, then exported everything to Word as a storyboard for developers to build the final version in Storyline.

Planning a Scenario

Before I sit down to write a scenario, I always know my objectives. What are you teaching or assessing?

I usually have an idea of how long the ideal or perfect path will be. If you have a multi-step process, that’s your ideal path. If there’s going to be 4 decision points on the shortest path, I know what those are before I start writing.

I also usually know at least some of the decision points based on errors or mistakes I need to address.

There’s a limit to how much you can plan before you just start writing it out though. I find it’s easier to just open up Twine and figure it out within that system.

Allow Opportunities to Fix Mistakes

One trick for managing the potentially exponential growth is by giving learners a chance to get back on the right path if they make a minor error. If they make 2 or 3 errors in a row, they get to an ending and have to restart the whole thing.

For example, maybe you’re teaching a communication skill where they should start with an open-ended question before launching into a sales pitch. Choice A is the open-ended question (the best choice). Choice B is a closed question (an OK choice). Choice C is jumping right into the sales pitch without asking (bad choice). After the customer response for choice B, I’d give them an opportunity to use the open-ended question (A) as their follow up. Reusing some choices helps keep it from growing out of control. In this image, reusing choices cuts the total number of pages from 40 to 20.

Flowchart for branching scenario with 20 pages Make Some Paths Shorter

Not every path needs to be the same length. In the above image, one branch from choice C is shorter. It ends after 2 choices instead of 3. You might make a short path if people make several major errors in a row. Past a certain point, it makes sense to  ask people to reset the scenario from the beginning or backtrack to a previous decision.

Good, OK, and Bad

In branching scenarios, not everything is as black and white as a clear-cut right or wrong answer. You can have good, OK, and bad choices and endings. In this example from my portfolio, green is good choices/endings, orange is OK choices/endings, and red is bad choices/endings. In this scenario, if you choose 39 (bad), you have 3 options: 40 (back on the good path, recovering from the mistake), 41 (OK), and 42 (a bad choice leading to a restart). This example has 15 endings, which is still more than I would like; if I was redoing it now I would probably collapse a few more of those endings together.

Branching scenario flowchart with good, OK, and bad choices and endings

Your Ideas?

Do you have any suggestions or tips for managing and reducing the complexity of branching scenarios? Please share in the comments.

Instructional Design Isn’t Dying. It’s Evolving.

You may have read dire predictions that instructional design is dead. The eLearning Guild just published a report titled “Is instructional design a dying art?” One of the guild’s recent surveys asked participants if ID is a dying field. Is it really?

Recently emerged monarch butterfly

No, It’s Not Dying; It’s Evolving

Instructional design is not dead or dying. That’s clickbait. This is a perennial hand wringing exercise. Marc Rosenberg wrote about it in 2004, and even 13 years ago he mentioned that this pops up every few years.

Instructional design isn’t dying; it’s evolving. Instructional design previously evolved from only classroom training to classroom plus online training. Now the field continues to evolve and expand. In fact, in the Guild report mentioned above, all 13 industry thought leaders agreed that instructional design is changing rather than dying.

As the field evolves, the name may change from instructional design to learning design, learning experience design or something else. I now call myself a “learning design consultant” rather than instructional designer. Regardless of the name, the core skills of instructional design will continue to be valuable and needed in the workplace.

Fragmentation and Diversification

I think instructional design will continue to fragment and diversify. Formal training isn’t disappearing; workers have too many skills they need and switch careers too often. In fact, I think ongoing formal training may even increase. Formal training will be accompanied by more informal training and performance support.

We will continue to have more potential skills than any single person can learn, so we will work more often in teams with specialists in particular skills.

New Technology and More Options

New technologies will give IDs new options. New technology often won’t completely replace old technology, but old and new will exist side-by-side. Sometimes how we use older technology will change. When TV became prevalent, radio didn’t disappear, but we listen in our cars now. Physical books haven’t vanished due to ebooks, but how we buy them has changed. Our future will likely include computers, mobile, AR, and VR. VR will be fantastic in certain situations, but it’s not going to be the right solution for every learning need.

Karl Kapp has noted that new technology is one reason the job outlook for instructional designers is still good.

Google Trends

One reason for the concern about instructional design dying is that it has been trending down on Google for a number of years. The trend has mostly flattened out, but it is much lower than it was in 2004. Brent Schlenker observed this trend in 2016.

Google Trends for "instructional design" 2004-2017

If you compare instructional design and learning design (the red line), you’ll see that learning design is now searched more often than instructional design. Learning design has also trended down, but not quite as far as instructional design.

Google trend comparing "instructional design" and "learning design"

While instructional design and learning design have trended down, elearning is trending up. I don’t believe the interest in online learning overall is likely to diminish, although it will evolve. Traditional self-paced elearning may decrease, but not all online learning.

 

Google trends showing elearning increasing

Looking Ahead

I see a fairly rosy future for instructional designers and learning designers, especially those who focus on lifelong learning and reflective practice. We will have to evolve to continue to be successful, but that need to constantly learn is part of what makes this field so rewarding. We will have to give up some of our old ways, but we can learn to change and be amazing.

“How does one become a butterfly?” she asked.
“You must want to fly so much that you are willing to give up being a caterpillar.”

—Trina Paulus, Hope for the Flowers

What do you think? Is instructional design doomed, or will we survive in a changed form?

 

 

Save