Book Review: Performance-Focused Smile Sheets

On a scale from 1 to 5, how useful are your current level 1 evaluations or “smile sheets”?

  1. Completely worthless
  2. Mostly worthless
  3. Not too bad
  4. Mostly useful
  5. Extremely useful

Chances are, your training evaluations aren’t very helpful. How much useful information do you really get from those forms? If you know that one of your courses is averaging a 3.5 and another course is averaging a 4.2, what does that really mean? Do these evaluations tell you anything about employee performance?

Personally, I’ve always been a little disappointed in my training evaluations, but I never really knew how to make them better. In the past, I’ve relied on standard questions used in various organizations that I’ve seen over my career, with mixed results. Will Thalheimer’s book Performance-Focused Smile Sheets changes that by giving guidelines and example questions for effective evaluations.

smile_sheets

Raise your hand if most of your evaluation questions use Likert scales. I’ve always used them too, but Thalheimer shows in the book how we can do much better. After all, how much difference is there between “mostly agree” and “strongly agree” or other vaguely worded scales? What’s an acceptable answer–is “mostly agree” enough, or is only “strongly agree” a signal of a quality course?

The book starts with several chapters of background and research, including how evaluation results should correspond to the “four pillars of training effectiveness.” Every question in your evaluation should lead to some action you can take if the results aren’t acceptable. After all, what’s the point of including questions if the results don’t tell you something useful?

The chapter of sample questions with explanations of why they work and how you might adapt them is highly useful. I will definitely pull out these examples again the next time I write an evaluation. There’s even a chapter on how to present results to stakeholders.

One of the most interesting chapters is the quiz, where you’re encouraged to write in the book. Can you identify what makes particular questions effective or ineffective? I’d love to see him turn this book into an interactive online course using the questions in that quiz.

I highly recommend this book if you’re interested in creating evaluations that truly work for corporate training and elearning. If you’re in higher education, the book may still be useful, but you’d have to adapt the questions since the focus is really on performance change rather than long-term education.

The book is available on Amazon and on SmileSheets.com. If you need a discount for buying multiple copies of the book, use the second link.

 

Save

Protagonists Should Be Like Your Learners

When you write a story for learning, you need a few essential elements such as a protagonist (the main character), the protagonist’s goal, and the challenges the protagonist faces. The protagonist should be someone your learners identify with. In workplace learning, that means the character has the same or a similar job as the learners. The learners should recognize the problems the character is dealing with and ideally share the protagonist’s goal. Learners should see a bit of themselves in the characters in your scenarios so they can picture themselves making the same kinds of decisions. When learners identify with your protagonist, they care about what happens to the character. They may be emotionally invested in seeing the protagonist succeed, especially in complex scenarios.

Joan is designing her first branching scenario

Example Protagonist Selection

Let’s review an example. Joan is an instructional designer working on a branching scenario. She has designed and developed many courses in the past, but this is the first time she has used a nonlinear format. She’s feeling a little nervous about getting it right. She’s creating training for front-line managers on how to handle requests for reasonable accommodations for disabilities. Which character should Joan use as the protagonist for her scenario?

  1. Mark, a technical writer with mobility issues who requires assistive technology
  2. Luisa, the VP of HR and an expert in accessibility issues
  3. Cindy, a manager with a team of 8 direct reports

Feedback on Your Choice

Mark would be a good choice for protagonist if this course were for employees to learn how to request reasonable accommodation. Someone like Luisa might be your SME for a course, but she has much more expertise than Joan’s learners. Cindy is a manager, which puts her in the same role as the learners. Joan will be able to put Cindy into situations similar to those managers might encounter. That will allow the learners to practice making the kinds of decisions they need to make in their jobs.

Other Characteristics

Joan might also be able to give Cindy other characteristics that make her similar to her learners. As part of her needs analysis, Joan interviewed two managers who had been through the process themselves. Both managers expressed reluctance to consult HR with questions about accommodations, even in situations where that was the best decision. Joan decides to create an option in the branching scenario where Cindy tries to handle the problem herself without HR but causes a costly misstep. Joan builds into the scenario the possibility of checking with HR before each decision and rewards that action with points in the final score.

Example compliance training with options to look up information

(This example scenario is also used in Motivating Learners to Look Up Compliance Policies Themselves.)

Joan3Stepping Back

At the risk of getting a little too meta, think back to Joan, the instructional designer. When you read that she was nervous about creating her first branching scenario, did that strike a chord with you? If you’re thinking about how to create your first scenario, that probably resonates. Even if you have created many branching scenarios in your career, you might still remember what it felt like to be unsure of yourself. If you’re an ID, you can probably envision yourself in this scenario. That gives you a connection to the character and helps engage you. You as the reader want to pick the right protagonist in the example so Joan’s course will be successful.

Characters in Cultural Context

Keep the culture of the workplace in mind as well. Your protagonist and other characters should reflect the organizational culture. In his report Using Culturally, Linguistically, and Situationally Relevant Scenarios, Dr. Will Thalheimer recommends:

In simulating workplace cues, consider the range of cues that your learners will pay attention to in their work, including background objects, people and their facial expressions, language cues, and cultural referents…

Utilize culturally-appropriate objects, backgrounds, actors, and narrators in creating your scenarios. Consider not just ethnicity, but the many aspects of culture, including such things as socio-economics, education, international experience, immersion in popular culture, age, etc.

Thalheimer’s recommendations from his research are also summarized on his blog.

How have you made your protagonists similar to your learners? Have you ever seen an attempt at scenario-based learning that was unsuccessful because the learners couldn’t identify with the main character?

Image credits

Debunker Club Works to Dispel the Corrupted Cone of Learning

A new group called The Debunker Club is working to dispel myths and misinformation in the learning field. From their website:

The Debunker Club is an experiment in professional responsibility. Anyone who’s interested may join as long as they agree to the following:

  1. I would like to see less misinformation in the learning field.
  2. I will invest some of my time in learning and seeking the truth, from sources like peer-reviewed scientific research or translations of that research.
  3. I will politely, but actively, provide feedback to those who transmit misinformation.
  4. At least once a year, I will seek out providers of misinformation and provide them with polite feedback, asking them to stop transmitting their misinformation.
  5. I will be open to counter feedback, listening to understand opposing viewpoints. I will provide counter-evidence and argument when warranted.

This year, coinciding with April Fool’s Day 2015, the Debunker Club is running an experiment. We’re making a concerted effort to contact people who have shared the Cone of Experience (also known as the Cone of Learning or the Pyramid of Learning).

Many iterations of this cone exist. A Google image search for “cone of learning” returns dozens of results, most of which are false. If you’ve seen something like this that said, “People remember 10% of what they read, 20% of what they hear, 30% of what they see” etc., you’ve seen a variation on this theme.

Image search results for Cone of Learning

The original cone was developed by Edgar Dale and didn’t include any numbers. The later versions are the “corrupted cone” with fictitious statistics added.  Will Thalheimer’s post from 2009 debunking these claims is where I learned it was incorrect. Common sense might give you a hint that these numbers aren’t really based in research though. Think about it–how many times have you see research where all the categories broke into even 10% segments?

As part of the Debunker Club’s efforts, I discovered a post on Dane’s Education Blog called A Hierarchy of Learning. Although this post cites a great debunking article (Tales of the Undead…Learning Theories: The Learning Pyramid), the blog author only says that he “appreciate what it conveys.”

I left the following comment on his Learning Pyramid post.

Thanks for collecting so many resources on your blog. I can see that you’ve worked really hard to share many links and ideas with your readers.

However, the information above, though it may appear to have scientific support, has been exhaustively researched and found to have no basis in science. In fact, the “Tales of the Undead” link you cite debunks it.

An article from the scientific journal Educational Technology shows no research backing for the information. (Subramony, D., Molenda, M., Betrus, A., and Thalheimer, W. (2014). The Mythical Retention Chart and the Corruption of Dale’s Cone of Experience. Educational Technology, Nov/Dec 2014, 54(6), 6-16.)

The information presented is likely to produce more harm than good, promoting poor learning designs and hurting learners.

While we might abstract some beneficial notions from the percentages portrayed in the misleading information — namely that encouraging realistic practice has benefits — there are numerous faulty concepts within the bogus percentages that can do real harm. For example, by having people think that there are benefits to seeing over hearing, or hearing over reading, we are sending completely wrong messages about how learning works.

Most importantly, recent advances in learning science have really come together over the last two decades. The misleading information was first reported in 1914, with no research backing. It’s better to follow more recent findings than information that has no scientific basis. See, for example, the book Make it Stick by Brown, Roediger, and McDaniel. Julie Dirksen’s Design for How People Learn is another great selection.

I’m part of a larger community of folks called the Debunker Club who are attempting to encourage the use of proven, scientifically-based learning factors in the learning field.

I’m going to be posting about this misleading information on my blog. I hope you’ll comment and respond to my post if you wish. I (and the debunker community in general) want to learn how other people feel about the issues and ideas surrounding the original information and our approach to debunking myths and sharing evidence.

If you’re interested in dispelling misinformation and improving the learning field, please join the Debunker Club and participate in the conversation.

Revisiting Learning Styles

As part of David Kelly’s Learning Styles Awareness Day, I’m revisiting the idea of learning styles. I admit that when I was taught learning styles in my education program, I didn’t question it. It made intuitive sense, and I’d never heard a real criticism of the theory. When I started digging into the research though, I realized that the research support for learning styles is pretty flimsy.

rhythm on a whiteboardIf I think back to the way learning styles were taught to me though, it was never applied the way that the theory is “officially” supposed to work. The most common idea is that people have some sort of style, and if you match that style they will learn better. That’s what Will Thalheimer’s still-unanswered research challenge asks for: something where individuals receive training matched to their style. If you’re a visual learner, you would only receive learning via visual methods; if you’re an auditory learner, you’d listen to everything you learn, etc.

That was never how it was applied in the classroom though. For K-12 classroom applications, learning styles were really about providing multiple methods of learning for everyone in the class. In a physical classroom, you didn’t have the option of individualizing everything, so you tended to look for ways to hit the visual and auditory at the same time or for multiple activities to reinforce the same content.

As a music teacher, that might mean something like teaching rhythms through multiple channels. I’d start by having students listen to me chant and clap a rhythm (auditory), then have them echo that rhythm back (auditory and kinesthetic). After several minutes of echoing rhythms with a specific type of pattern, I’d draw a rhythm on the chalkboard (yes, actual chalk) and connect how it looks to how it sounds (visual and auditory). Then we’d practice reading some rhythms with similar patterns, with them looking, chanting, and clapping all together.

If I was teaching music today, I’d do that same kind of lesson, just not because of learning styles. That’s all based on the Kodály method, which does have research support (at least as far as I know; I haven’t dug into it since I rarely teach music anymore). But the idea of approaching concepts from multiple angles with different methods and media still makes sense. It isn’t because I’m matching to a particular style; it’s because I’m helping everyone learn through multiple channels. This might be what Tom Stafford from Mind Hacks is getting at when he says “Having thought about learning styles helps teachers improve their teaching and also helps increase their confidence and motivation.” I really wish he provided a citation for the idea that thinking about learning styles helps teachers improve their teaching though; I’d like to know whether that’s just his opinion or something with data to support it.

So what does this mean for me as an instructional designer today, rather than a K-12 music and band teacher? As an instructional designer, I basically ignore learning styles. I do think about presenting information with both visuals and audio, but that’s more based on cognitive load theory than learning styles. I’m also working to do better at visual presentation with graphics and not just words, because that is supported by research. As Judy Unrein noted “…humans are such overwhelmingly visual creatures that if we simply catered better to that one sense, we could improve the vast majority of our designs.”

Judy’s idea of focusing on interaction preferences is an interesting one. People do have different preferences, and those preferences can change based on the context (and the type of content, I would add). Giving learners some control over how they interact with the training does seem beneficial. If we don’t lock down the navigation, they can choose which parts they really need. In spite of the research, I personally find audio in e-learning to be generally obnoxious, so if I can turn it off and read the captions instead, I almost always will do that instead. I can read much faster that you can read to me, thank you very much, so I’m annoyed if you don’t give me the option of reading.

What about you? Is there anything in learning styles that you find useful in your own practice, or is it something you’ve abandoned in favor of other ideas?

Image credit: rhythm by billaday