Tag: feedback

Using Time as Scenario Feedback

Nicole is creating a branching scenario practicing communication techniques for nutrition counselors to better understand their clients’ goals. She has written a simulated conversation between a counselor and a client. Her SME, Brian, provided this feedback after reviewing the prototype.

The conversation overall does a good job giving plausible choices for questions and showing realistic responses from the client. I want this to be really exciting for learners, like a game. Let’s add a timer for each decision. That way, they’ll be motivated to answer quickly and keep pushing through the scenario.

What do you think? How should Nicole respond to Brian?

A. That’s a great idea! I think that will enhance the learner experience.

B. I’m not sure. Let me do some research.

C. Timing might not be the best form of feedback for this particular course.

Remember your answer; we’ll come back to this question at the end of the post.

Using Time as Scenario Feedback

When Time is Effective

Time can be a very effective consequence in some learning situations. Check out the Lifesaver training on what to do in emergency situations for an example with effective use of time as feedback.

In the first scenario with Jake, you help someone in cardiac arrest. Each question has a 5 second timer, and you are scored for both accuracy and speed.

6/6 Right First Time - Avg Speed 1.32s

Later in the scenario, you simulate performing CPR by pressing two keys on your keyboard in the same rhythm as CPR. While you practice, you see a scale from good to bad showing how close you are to the ideal timing. This lets you adjust your rhythm. After you finish the 30 seconds of simulated CPR, you see a percentage score for your accuracy.

Scale showing good and bad for speed

This feedback works in the Lifesaver training because timing really is a critical part of the skill being taught. Speed of response matters in these emergency situations, as does knowing the right rhythm for CPR.

Time can work for other skills too, like manufacturing, making sandwiches in a chain restaurant, or safety training.

When Time is Counterproductive

If the skill you’re practicing and assessing requires critical thinking and careful consideration, measuring time can be counterproductive. For simulated conversations where you want learners to pause and think about their options, it’s better to not use a timer.

You might be thinking, “But in a real conversation, people need to think quickly. Doesn’t that mean we should use timers?” That’s a question about fluency, which requires more practice over time. If your goal is to get people to that point of fluency, you might add a timer, but not for the initial practice. Teach the skill without a timer first, then provide additional practice opportunities to build fluency and speed in the skill.

Do Timers Improve Motivation?

Does a timer increase motivation for getting the right answer? Maybe, if the learners are already motivated prior to starting and time makes sense in the context of the activity. Many games use time as a way to keep players engaged and excited.

I suspect in practice that unnecessary timers encourage people to guess randomly for whatever they can click fastest. Time may actually decrease learners’ motivation to be truly cognitively engaged with the learning experience. They may be more motivated just to click and get through it quickly than to read carefully and understand their decisions.

Accessibility

Timers can create challenges for accessibility. Learners with visually impairments who use a screen reader and keyboard navigation will generally need more time to answer. Learners with  mobility impairment may have trouble manipulating a mouse or keyboard quickly. Depending on your audience, adding timers may prevent some learners from being successful in your elearning courses, even if they could do the real task (like having a conversation) without problem.

Revisiting the Communication Scenario Example

Think back to Nicole’s scenario at the beginning of this post. She’s teaching communication skills to nutrition counselors, using a simulated conversation. Her SME, Brian, suggested adding a timer.

What do you think? Does a timer seem helpful in this situation?

Probably not. In this training, it’s more important for learners to think carefully about their choices and responses than to be speedy. Feedback like the expression of the client or a scale showing the client’s motivation to change their eating behavior would be more beneficial than feedback on how quick they are.

Your Examples?

Time can work as feedback in learning scenarios, but it should be used sparingly, and only when it is actually relevant to the skill being practiced or assessed.

Do you have any examples of time used successfully as feedback in a scenario? I’d love to see some more samples. Share them in the comments.

 

ID and eLearning Links (12/3/17)

  • This is the link I send people to debunk the blanket claims about “people forget X% after Y time.” The reality is that how much people forget depends on who your audience is, what they’re learning, and how you train them.

    tags:training instructionaldesign research myth

    • The amount a learner will forget varies depending on many things. We as learning professionals will be more effective if we make decisions based on a deep understanding of how to minimize forgetting and enhance remembering.

    • To be specific, when we hear statements like, “People will forget 60% of what they learned within 7 days,” we should ignore such advice and instead reflect on our own superiority and good looks until we are decidedly pleased with ourselves.

    • Many of the experiments reviewed in this report showed clearly that learning methods matter. For example, in the Bahrick 1979 study, the best learning methods produced an average forgetting score of -29% forgetting, whereas the worst learning methods produced forgetting at 47%, a swing of 76% points.

  • Mini-scenarios and branching scenarios provide better assessment than traditional multiple choice, but this provides some other options for deeper assessment that can still be scored by a computer.

    tags:assessment scenario e-learning instructionaldesign feedback

Posted from Diigo. The rest of my favorite links are here.

Instructional Design and E-Learning Links

What to Write First in Branching Scenarios

Writing a branching scenario can be intimidating or overwhelming. Unlike a linear course, it’s not as easy to know where to start writing. Do you write the endings first? Do you write all the mistakes first? Do you start at the beginning and then flesh out each path as you write those choices?

I have found that it’s easiest to write the ideal path from start to finish first. I note decision points and sometimes draft bad choices along the way, but I don’t fully write anything else until I finish the ideal path.

What to Write First in Branching Scenarios

Writing the Ideal Path from the Outline

In my last post, I explained how I create an outline of the steps in the scenario. This is my plot outline for the story if learners take the “ideal path,” making the best decision at every step. For each step in the outline, create a situation in which the learner must choose to take that action. You create a decision point where the best choice is the first step in your outline.

Write the First Decision

Building on the example from my last post, a course on screening potential consulting clients, I have a process with 4 steps.

  1. Send client initial screening questions.
  2. Review client responses for fit and feasibility.
  3. Learn more about client needs during preliminary phone call.
  4. Propose a short road mapping engagement.

Because I did that planning in advance, I know exactly what I’m writing first: a decision where the right choice is sending the client initial screening questions.

Sophie receives an email from a prospective client, Robert.

Hello Sophie,

My company has 4 classroom training courses we’d like to convert to online. One of them is a half day course; the others range from one day to four days long. Can you please tell me what you would charge to convert these courses to online?

Regards,

Robert

What should Sophie do?

    1. Send Robert a price estimate.
    2. Send Robert some client screening questions.
    3. [[Some other OK choice TBD]]

Write the Remaining Ideal Decisions and Consequences

Once you have the first step written, the next thing you will write is the consequence from making that best decision in step one. In this example, the prospective client will reply to the email.

Robert replies with his answers to the screening questions.

[placeholder–questions and answers here]

What should Sophie do?

  1. These answers look reasonable. Schedule a call to discuss it further.
  2. [[OK choice TBD]]
  3. [[Bad choice TBD]]

Continue writing until you get to the end of the ideal path, showing the consequences for good decisions and how they lead to the next decision.

Don’t Write the Mistakes Yet

When I write my initial draft of the ideal path, I focus just on the correct or best choices first. I don’t write all of the mistakes and their consequences on the first pass through writing. As I draft choices, I might write down some of the bad choices if I already know them. For example, in step one, I know the mistake I’m trying to avoid is the first choice above of sending a price estimate without understanding the problem and scope first. However, at this stage of writing, it’s OK to just leave a placeholder for the other choices.

I find it helpful to indicate what kind of choices I still need in the placeholder. For most scenarios, the majority of decision points have a Good, OK, and Bad choice.  You can see how I noted that in my placeholders as “OK choice TBD” or “Bad choice TBD.”

Write the Ideal Ending and Feedback

At the end of the scenario, after learners have made all the correct decisions, write the ending. This should show the positive consequences of those choices. The ending should show what it looks like when people meet the learning objectives. In this example, the ending will show Sophie and Robert working together with a productive relationship.

You may also choose to include some more instructional feedback or coaching. At the end of the scenario, it can be helpful to tell people why the decisions they made were correct to reinforce what they learned.

Use Twine for Writing

I have tried a number of different tools and methods for writing branching scenarios. The open source tool Twine is my favorite for writing scenario drafts and creating quick prototypes. This makes it easy to see the connections between decision points. It’s also easy to leave placeholders for other choices that you will flesh out later.

More Reading

In my next post, I’ll describe how I write the mistakes and flesh out the rest of the scenario.

You might also be interested in my other posts on branching scenarios.

 

Book Review: Practice and Feedback for Deeper Learning

Patti Shank’s Practice and Feedback for Deeper Learning is a summary of tactics you can use to create memorable, relevant practice opportunities and provide constructive, beneficial feedback for learners. Everything in the book is backed by research and written to be immediately usable by instructional designers and trainers.

Cover: Practice and Feedback for Deeper Learning

This is the second installment in Patti’s “Make It Learnable” series, which is shaping up to be one of those sets of fundamental reading in the field of instructional design. The first book is Write and Organize for Deeper Learning; you can read my review of the first book. As with that book, this book gives you a shortcut to what really works based on evidence, without having to wade through complex (and often contradictory) research yourself. Specifically, this is based on training research, not research on K-12 or higher education learners.

Have you ever wondered…?

  • How do we create practice activities that will help transfer skills to the workplace?
  • Ho can we create practice activities that are more memorable?
  • How can we create more effective feedback than just “correct” and “incorrect”?
  • Do novice and experienced learners benefit from the same strategies?
  • How do we make sure learners are practicing the right skills and behaviors?
  • How can we help learners deal with errors and mistakes?
  • If we’re training a complex task, should we divide the task into small parts or train a simple version of the whole task?
  • Is it better to give feedback right away or to delay it?
  • What kinds of realism are important to training practice? Is it necessary to use lots of multimedia to make training look exactly like the work environment?
  • Is it better to set goals for specific performance levels or goals for making progress in learning?

All of these questions are addressed in this book through 5 overall strategies divided into 26 tactics.

Go buy Practice and Feedback for Deeper Learning now. Read it, and then pick something relevant to apply to your own work. After all, the best way to improve your own learning design is to practice using these tactics yourself.

Immediate and Delayed Consequences in Branching Scenarios

In branching scenarios, we can use a combination of immediate and delayed consequences and feedback. Consequences are what happens as a result of decisions; feedback is what we tell learners after decisions.

Immediate & Delayed Consequences

Use Immediate Consequences Often

Immediate consequences are the intrinsic effects of decisions. A customer who responds angrily, software that doesn’t produce the desired result, or time lost on a project could all be immediate consequences. These consequences don’t directly tell the learner, “Sorry, that was incorrect.” Learners have to perceive and understand the cues in the scenario. They have to draw conclusions based on those cues.

If your learners will need to follow cues when they apply what they’re learning, it’s helpful to provide real-world consequences in your scenario. It’s beneficial to practice interpreting cues.

Immediate consequences that simulate real-world cues can also be more engaging than the omniscient narrator dictating what you did right or wrong.  It keeps learners in the mindset of the story without hitting them over the head with a reminder that they’re learning something.

Use Immediate Feedback with Novices

Immediate feedback is different from intrinsic consequences. This is the instructional feedback or coaching that directly tells learners why their decisions are right or wrong. While this can pull people out of the “flow” of a story, immediate feedback can be helpful in some situations.

First, novice learners who are still building mental models of a topic may benefit more from immediate feedback. Novices may not have the expertise to sort through real-world cues and draw accurate conclusions from them.  Therefore, it may be more important to provide immediate feedback after each decisions in a branching scenario if your audience is new to the topic.

In his research report “Providing Learners with Feedback,” Will Thalheimer explains the benefits of immediate feedback for novices.

“On the surface of it, it just doesn’t make sense that when a learner is piecing together arrays of building blocks into a fully-formed complex concept, they wouldn’t need some sort of feedback as they build up from prerequisite concepts. If the conceptual foundation they build for themselves is wrong, adding to that faulty foundation is problematic. Feedback provided before these prerequisite mental modelettes are built should keep learners from flailing around too much. For this reason, I will tentatively recommend immediate feedback as learners build understanding.”

Provide Instructional Feedback Before a Retry

I always use feedback before restarting a scenario.  If a learner has reached an unsatisfactory ending in a scenario, it’s beneficial to do a short debrief of their decisions and what went wrong. Especially for more experienced learners, some of that feedback may be delayed from when they made the decision. You can summarize the feedback for several previous decisions on the path that led to the final decision.

This feedback should happen before they are faced with the same scenario decisions again. Otherwise, they could make the same mistakes again (reinforcing those mistakes) or simply guess without gaining understanding.

Thalheimer’s research also supports this.

“When learners get an answer wrong or practice a skill inappropriately, we ought to give them feedback before they attempt to re-answer the question or re-attempt the skill. This doesn’t necessarily mean that we should give them immediate feedback, but it does mean that we don’t want to delay feedback until after they are faced with additional retrieval opportunities.”

Use Delayed Feedback with Experienced Learners

Thalheimer notes that delayed feedback may be more effective for retention (i.e., how much do learners remember). That effect might be due to the spacing effect (that is, reviewing content multiple times, spaced out over times, is better for learning than cramming everything into a single event). The delay for feedback doesn’t have to be long; one study mentioned in Thalheimer’s report showed that delaying feedback by 10 seconds improved outcomes.

Delayed feedback may also be more appropriate for experienced learners who are improving existing skills rather than novices building new skills. Experienced learners already have mental models in place, so they don’t have the same needs for immediate correction as novices. They can get the benefit of delayed feedback.

Use Delayed Feedback with Immediate Consequences

In branching scenarios, we can use a combination of immediate intrinsic consequences (e.g., an angry customer response) and delayed instructional feedback (e.g., you didn’t acknowledge the customer’s feelings). Feedback before a retry or restart could count as delayed if it includes feedback for multiple decisions. If you let learners make 2 or 3 wrong choices before a restart, the combined feedback will effectively be delayed.

Use Delayed Consequences When Realistic

We  don’t always immediately know our mistakes are wrong in real life. Sometimes the consequence isn’t obvious right away. Sometimes it seems like a gain the short run, but causes problems in the long run. If that’s the kind of situation you’re training for, letting people continue on the wrong path for a little while makes sense. Neither limited branching nor immediate failure allow you to show delayed consequences.

Providing these delayed consequences has the advantage of better learning from delayed feedback, plus it creates a more realistic and engaging story. Delayed consequences shouldn’t be forced into a scenario where it’s not realistic, but they are a good way to show the long-term effects of actions.

Think about how delayed consequences could be shown in these examples:

  • A bartender gives away many free drinks. The immediate consequence is that the customers are happy, but the delayed consequence is a loss of profit for the bar.
  • A sales associate sells a customer a product that is less expensive but meets the customer’s needs. The immediate consequence is that the sales associate makes less commission that day, but the delayed consequence is that the customer is loyal and refers 2 friends. In this case, the total commission earned is higher even though the immediate sale was lower.
  • A doctor could skip a screening question with a patient. The immediate consequence is finding something that looks like the problem, but the delayed consequence is the actual underlying problem remaining.
  • A manager asks an ID to create training. The ID gets started building it right away, trusting that the team requesting the training knows their needs. The immediate consequence is a happy manager, but the delayed consequence is ineffective training that doesn’t actually solve the business problem.
  • If you’re teaching ethics, a small ethical lapse early in the scenario might not seem like a big deal. The immediate consequence might be meeting a deadline or increased recognition.  In the long run, that small lapse leads a continued need to cover up your actions. The Lab: Avoiding Research Misconduct is an example with delayed consequences in some paths.

Looking for More?

Read more about branching scenarios: