Tag: Ruth Clark

Feedback in Branching Scenarios: What Works for Novices, Experts, and Everyone

When we provide feedback in branching scenarios, we have several questions to consider.

  • Should we provide consequences (intrinsic feedback) or coaching (instructional feedback)?
  • Should we provide immediate feedback or delayed feedback?
  • What works for novices versus experts?

Intrinsic and Instructional Feedback

In Scenario-based e-Learning: Evidence-Based Guidelines for Online Workforce Learning, Ruth Clark recommends combining intrinsic and instructional feedback.

Intrinsic feedback is the consequences for an action. It’s what happens because of the learner’s decisions. If you have a scenario where an employee falls off a ladder, a customer agrees to buy a more expensive product, or a patient recovers from a medical emergency, that’s intrinsic feedback. You show the learner what happens.

Instructional feedback is coaching that tells the learner about their choice rather than showing them. In a branching scenario, instructional feedback could come from a coach or character that guides learners. Instructional feedback doesn’t necessarily have to mean telling people directly if their choice was correct or incorrect. Learners should be able to figure that out from the intrinsic feedback. Instead, instructional feedback can focus on addressing misunderstanding or explaining why a choice had a certain result.

Novices may need more instructional feedback than experts. Experts are less likely to have problems with cognitive load from sorting through multiple pieces of information in a scenario. Experts are better at diagnosing their own problems based on contextual information like intrinsic feedback. Novices, on the other hand, may need more direct coaching to make sense of the intrinsic feedback, especially when they fail a scenario.

Immediate and Delayed Feedback

When we build branching scenarios, immediate consequences provide realism and keep learners engaged. Every time learners make a decision, something happens: the customer responds, the equipment breaks, or sales go up.

Note that “immediate” here refers to when the learner receives the feedback, not how quick the results would happen in real life. If a learner makes a choice to ignore recommended equipment maintenance to save money, you could jump ahead in time three months to show that equipment breaking and costing more money in the long run. As long as you show the feedback right away, it’s immediate because it gives learner information about their choice immediately.

Delayed consequences happen in branching scenarios when you show one consequence immediately, but a different consequence appears later.

For example, let’s take a scenario where a manager asks an ID to create training. The learner chooses to have the ID start building it right away, trusting that the team requesting the training knows their needs without further analysis.

  • The immediate consequence is that the ID’s manager is happy.
  • The delayed consequence is that the ID creates ineffective training that doesn’t actually solve the business problem.

You can also use delayed feedback, or coaching delivered to the learner later. In his report on Providing Learners with Feedback, Will Thalheimer suggests that feedback should be provided before learners try again. While that research was more related to retaking tests, I think that’s a good guideline for scenario-based learning. If learners fail a scenario and are asked to try again, give them some feedback to help them learn from their mistakes and make better choices next time.

Novices may benefit from more immediate feedback and coaching, while experts may be fine just receiving coaching at the end of a scenario.

Recommendations for Feedback

Here are my overall recommendations for feedback in scenario-based learning. These are based on a combination of research reviews from Clark and Thalheimer, along with recommendations from Cathy Moore, Michael Allen, and others, plus my own experience.

scenario_feedback

For Everyone

  • Provide frequent, immediate consequences that show learners what happens as a result of their decisions.
  • Provide coaching before learners retry a scenario.
  • Use delayed consequences in scenarios where they are realistic, although note that novices may need more coaching to help them understand delayed consequences.

For Novices

  • Provide immediate coaching for novices, especially to correct misconceptions or incorrect strategy selection.

For Experts

  • Use more delayed coaching with expert learners.

Don’t Assume the Recommendations are Perfect

None of these recommendations are correct 100% of the time for every situation or every group of learners. I’m fairly confident recommending frequent immediate consequences and coaching before a retry, but you may find exceptions even to those recommendations. The research on feedback is sometimes contradictory, so there is little firm guidance.

To quote Will Thalheimer, describing conflicting research results, “First, it tells us that we should be skeptical of absolutism. In particular, it would be perilous for us to say, ‘Immediate feedback is always better,’ or, ‘Delayed feedback is always better.'”

Let’s use the research to guide our decisions in providing feedback, but let’s also acknowledge that the research has limitations. Sometimes we have to use our best judgement on how to best support our learners.

 

Benefits of Scenario-Based Learning

Why are scenarios effective for learning? They provide realistic context and emotional engagement. They can increase motivation and accelerate expertise. Here’s a selection of quotes explaining the benefits.

Benefits of Scenario-Based Learning

Accelerating Expertise with Scenario-Based e-Learning – The Watercooler Newsletter : The Watercooler Newsletter: Ruth Clark on how scenario-based elearning accelerates expertise and when to use it

What is Scenario-Based e-Learning?

  1. The learner assumes the role of an actor responding to a job realistic situation.
  2. The learning environment is preplanned.
  3. Learning is inductive rather than instructive.
  4. The instruction is guided.
  5. Scenario lessons incorporate instructional resources.
  6. The goal is to accelerate workplace expertise.

As you consider incorporating scenario-based e-Learning into your instructional mix, consider whether the acceleration of expertise will give you a return on investment.  For example, interviews with subject matter experts indicated that automotive technicians must complete about 100 work orders to reach a reasonable competency level in any given troubleshooting domain.  Comparing delivery alternatives, OJT would require around 200+ hours, instructor-led training would require around 100 hours, and scenario-based e-Learning simulations require approximately 33–66 hours.

Finally, many learners find scenario-based e-Learning more motivating than traditional instructional formats.  Solving a work-related problem makes the instruction immediately relevant.

The Benefits of Scenario Based Training: Scenario-based training better reflects real-life decision making

There is no linear path into what they are subjected. The situations are complex. They often fail and they learn by reflection, becoming much better at the judgements they make next time, even though next time the environment and the scenarios presented are different.

After completing a few exercises, they build their own view of the patterns that are evident and are able to move into a new scenario with confidence even if the environment and scenario is radically different.

Learning on reflection before plunging into the next scenario helps to build the patterns in the participants’ minds that are the evidence that they have learnt.

Quizzes based on scenarios with a, “What would you do next?”, question builds quick and fun repetition into the training programme, helping transfer from short term memory to long term memory.

Scenario-based-learning: PDF explaining theory and how to decide if SBL is the right strategy

Scenario-based learning is based on the principles of situated learning theory (Lave & Wenger, 1991), which argues that
learning best takes place in the context in which it is going to be used, and situated cognition, the idea that knowledge is
best acquired and more fully understood when situated within its context (Kindley, 2002).

SBL usually works best when applied to tasks requiring decision-making and critical thinking in complex situations. Tasks
that are routine to the students will require little critical thinking or decision-making, and may be better assessed using
other methods.

Checklist: Is SBL the right option? (Clark, 2009)
* Are the outcomes based on skills development or problem-solving?
* Is it difficult or unsafe to provide real-world experience of the skills?
* Do your students already have some relevant knowledge to aid decision-making?
* Do you have time and resources to design, develop, and test an SBL approach?
* Will the content and skills remain relevant for long enough to justify the development of SBL?

Learning through storytelling | Higher Education Academy: Why storytelling works for learning

Stories are effective tools for learning due to their ability to facilitate the following cognitive processes: i) concretizing, ii) assimilation, and iii) structurizing (Evans and Evans 1989).

Top 7 Benefits You Get From Scenario-Based Training: Infographic on benefits. “Falling forward,” accelerating time, critical thinking, shared context, engaging emotions, retention, trigger memories

Scenarios allow “falling forward”: Providing a safe space to fail helps build the capacity to fix mistakes as you would in real life

When Is Audio Narration Helpful?

In a discussion on eLearning Heroes, Judith Reymond asked about the research on when or whether audio narration is helpful to adult learners.

Speaker and sound waves

In Clark and Mayer’s eLearning and the Science of Instruction, they say that the research generally supports using narration with on-screen visuals. Adult learners retain more from a narration plus visuals approach than from reading on-screen text. They call this the “modality principle.”

Generally speaking, when you have narration, you shouldn’t also have that same text on the screen. This is called the “redundancy principle.” Clark and Mayer note some exceptions when text should be shown on screen (pp. 87-88, 107-108 in the 1st ed):

  • Complex Text: Complex text like mathematical formulas may need to be both on-screen and in narration to aid memory. (In practical experience, I also do this for text that has to be memorized word for word, such as screening questions for addiction.)
  • Key Words: Key words highlighting steps in a process or technical terms
  • Directions: Directions for practice exercises. “Use onscreen text without narration to present information that needs to be referenced over time, such as directions to complete a practice exercise.”
  • No Graphics: When there are no graphics or limited graphics on the screen
  • Language Difficulties: When the audience has language difficulties. I have used redundant on-screen text for an audience with very low literacy and a high percentage of learners with English as a second language. It might be enough to simply provide a transcript or closed captions in those situations so people who don’t need it can ignore or turn off the text.

In practical terms, I’ve found that if every page has narration and you suddenly have no narration for a practice exercise, some learners think something’s broken on the page. I generally have the narrator say something short to introduce the practice exercise, but leave the directions as on-screen text.

However, it’s also tiring to listen to a voice. I usually don’t provide audio feedback on practice activities to give people a break. I’ll sometimes provide other kinds of interaction or content delivery to provide a break from the audio (tabs or “click to reveal” text).

In the book, Clark and Mayer say this:

“Does the modality principle mean you should never use printed text? Of course not. We do not intend for you to use our recommendations as unbending rules that must be rigidly applied in all situations. Instead, we encourage you to apply our principles in ways that are consistent with the way that the human mind works—that is, consistent with the cognitive theory of multimedia learning rather than the information delivery theory.”

The principle of avoiding redundant on-screen text is sometimes treated as sacrosanct. I’ve seen some big names in the field practically yell that this is a firm rule that should never be broken. In real life, it’s not as clear cut, as even Clark and Mayer acknowledge. There’s plenty of redundant on-screen text that has no business being there. You should be thoughtful and intentional if you’re going to provide on-screen text. Generally, it shouldn’t be there, and you need a real reason to break the redundancy principle.

What are your experiences with audio, especially with on-screen text? What have you found works with your audiences?

Intrinsic and Instructional Feedback in Learning Scenarios

A few years ago, I was a judge for a competition on scenario-based learning. While there were a few terrific submissions, I thought many of the courses missed the whole point of scenario-based learning. They started out fine: they provided some sort of realistic context and asked learners to make a decision. Then, instead of showing them the consequences of their decision, they just provided feedback as if it was any other multiple choice assessment. “Correct, that is the best decision.” Blah. Boring. And ineffective.

In her book Scenario-based e-Learning: Evidence-Based Guidelines for Online Workforce Learning, Ruth Clark labels the two types of feedback “intrinsic” and “instructional.” Instructional feedback is what we see all the time in e-learning; it’s feedback that tells you what was right or wrong and possibly guides or coaches you about how to improve.

With intrinsic feedback, the learning environment responds to decisions and action choices in ways that mirror the real world. For example, if a learner responds rudely to a customer, he will see and hear an unhappy customer response. Intrinsic feedback gives the learner an opportunity to try, fail, and experience the results of errors in a safe environment.

Intrinsic feedback is one of the features of scenario-based learning that sets it apart from traditional e-learning. When you show learners the consequences of their actions, they can immediately see why it matters. The principles or process that you’re teaching isn’t just abstract content anymore; it’s something with real world implications and it matters if they get it wrong. It’s more engaging to receive intrinsic feedback. Learners are also more likely to remember the content because they’ve already seen what could happen if they don’t make the right choices.

Intrinsic feedback can take a number of forms. Customer reactions (verbal and nonverbal), patient health outcomes improving, sales figures dropping, a machine starting to work again, and other environmental responses can be intrinsic feedback. The example below contains three pieces of intrinsic feedback, all on the left side: a facial expression, a conversation response, and a motivation meter at the bottom.

Screenshot of a branching scenario with intrinsic and instructional feedbackIn this example, learners are trying to convince someone to make healthier eating choices using motivational interviewing. Motivation level is an “invisible” factor, so I made it visible with a motivation indicator in the lower left corner. As learners make better choices and the patient feels more motivated to change, the motivation meter shows their progress.

Scenarios can also use instructional feedback. In the above example, a coach at the top provides instructional feedback and guidance on learners’ choices. Clark recommends using both intrinsic and instructional feedback in most situations.

One issue with instructional feedback is that it can break the realism of a scenario. Using a coach can help alleviate that problem, as can having learners ask for advice from people inside a scenario (a manager, an HR rep, another worker, etc.). Using a conversational tone for the instructional feedback also helps keep it within the scenario. Instructional feedback in a scenario often doesn’t need to explicitly say that a choice was correct or incorrect; that’s clear enough from the intrinsic feedback. Focus your instructional feedback on explaining why a choice was effective or how it could have been better.

Feedback can also be delayed rather than happening immediately. Clark recommends more immediate feedback for novices but delayed feedback for experts or more advanced learners. Depending on the audience, for some branching scenarios I do immediate intrinsic feedback for each choice learners make. When learners make bad choices that cause them fail and they need to restart the scenario, they receive instructional feedback with guidance on how to improve on their next attempt. They might be able to make two or three bad choices in a row before they hit a dead end in the scenario, so the instructional feedback is delayed. It keeps the momentum of the scenario moving forward but still provides support to learners to help them improve.

If you’re building scenario-based learning, don’t leave out the intrinsic feedback! Your learners will thank you.

12+ Books for Instructional Designers

If you’re looking for some reading to improve your skills or get started in the field of instructional design, check out these books.

ID books

General Instructional Design and E-Learning

Design For How People Learn (now in its second edition) by Julie Dirksen is one of my favorite books in the field. I’ve recommended it many times. It’s easy to read and understand. It makes research about learning accessible in ways you can apply immediately. The illustrations are charming and reinforce the concepts well. Read my review for more details.

The Accidental Instructional Designer by Cammy Bean is especially good for career changers and those who landed in instructional design from other fields. It provides a model for the range of skills that fall under the umbrella of “instructional design.” It includes practical tips on topics such as working with SMEs and avoiding “clicky clicky bling bling” or flashy interactivity and multimedia for the sake of being flashy. The design models in chapter 4 are probably familiar to many with experience in the field but very helpful to beginners who want to do more than just the same type of course and interaction for every situation.

Designing Successful e-Learning by Michael Allen tells you to “Forget What You Know About Instructional Design and Do Something Interesting.” All of Allen’s books are focused on helping people design e-learning that is interactive, engaging, and useful.

e-Learning and the Science of Instruction by Ruth Clark and Richard Mayer is one of the first books on e-learning I bought, and I still refer to it when I need evidence to justify decisions to clients. If you’ve ever wondered if formal or conversational style is better for learning (conversational) or if your on-screen text should replicate what’s on the screen (no, it shouldn’t), this book explains it with the research to back it up. It’s not perfect; the authors do sometimes disregard research that contradicts their own findings, and they sometimes make their principles seem more absolute than they probably are in real life. However, it’s still a solid reference.

First Principles of Instruction: Identifying and Designing Effective, Efficient and Engaging Instructionis David Merrill’s effort to distill the common principles from multiple instructional design theories. A shorter, earlier explanation of these principles is available as a free PDF.

Games and Scenario-Based Learning

The Gamification of Learning and Instruction by Karl Kapp explains how to do more with gamification than just badges and points. Karl summarizes research and game theory and explains how substantive elements of games like narrative can be used to improve learning design. I wrote more about this gamification research previously.

Scenario-based e-Learning by Ruth Clark is similar to eLearning and the Science of Instruction in that it summarizes research findings. This book is specifically focused on developing scenario-based e-learning, including everything from simple branching scenarios to complex simulations.

Learning Communities

Building Online Learning Communities by Rena Palloff and Keith Pratt is aimed more at online instructors than instructional designers, but it’s a wonderful resource for IDs working in higher education or supporting online and blended learning communities.

Digital Habitats: Stewarding Technology for Communities by Etienne Wenger, Nancy White, and John Smith is about how technology can enable communities of practice.

Other Topics

Learning Everywhere by Chad Udell is a fantastic resource on mobile learning, providing everything from a big picture view of broad categories of mobile learning to specific technical considerations and pitfalls. You can read my review of the book for more details.

Show Your Work by Jane Bozarth is full of visuals and explains how to “show your work” by sharing what you’re doing and learning using social tools. The book explains the benefits of creating a culture where people share their processes and discoveries.

Now updated to E-Learning Uncovered: Adobe Captivate 9, which I’m sure is just as good as the last edition. E-Learning Uncovered: Adobe Captivate 8 by Diane Elkins, Desiree Pinder, and Tim Slade sat on my desk for multiple weeks because I used it so often that it wasn’t worth bothering to put it back on the shelf. This book was an immense help to me in learning Captivate 8. I’m sure their other books on Storyline 2, Lectora, etc. are equally valuable.

More Reading Lists

I received so many great suggestions after posting this list that I posted 20+ More Books for Instructional Designers.

If that’s not enough, these reading lists will give you additional ideas.

Your Selections

Did I miss one of your favorite books? Leave a comment with your suggestions.