Ruth Clark Claims “Games Don’t Teach”

Ruth Clark posted at ASTD an article titled “Why Games Don’t Teach.” It’s a deliberately provocative title, meant to draw attention and cause controversy. A more accurate title would be “Some Games Aren’t Effective at Making People Remember Content,” but that’s a lot less likely to grab attention.

Before I continue, I want to say that I enjoyed her book, eLearning and the Science of Instruction, and I have found some of the research there valuable. I respect her past contributions to the field.

However, I think Clark didn’t do a very careful review of the literature before writing her post, and I don’t think that one study is enough for her to make such a broad claim dismissing games for learning.

Oregon Trail for the iPhone
According to Ruth Clark, you didn’t learn anything playing Oregon Trail, Carmen Sandiego or Lemonade Stand

Let’s look at her summary of the research:

The goal of the research was to compare learning efficiency and effectiveness from a narrative game to a slide presentation of the content. Students who played the Crystal Island game learned less and rated the lesson more difficult than students who viewed a slide presentation without any game narrative or hands on activities. Results were similar with the Cache 17 game. The authors conclude that their findings “show that the two well-designed narrative discovery games…were less effective than corresponding slideshows in promoting learning outcomes based on transfer and retention of the games’ academic content” (p. 246).

The research is behind a paywall, of course, but the abstract is online. (Update 8/11/2014: A copy of the original article can now be found outside the paywall.)

Adams, D.M., Mayer, R.E., MacNamara, A., Koenig, A., and Wainess, R. (2012). Narrative games for learning: Testing the discovery and narrative hypotheses. Journal of Educational Psychology, 104, 235-249.

Next, let’s look at how the authors summarize their own work and see how it compares to Clark’s summary (emphasis mine).

Overall, these results provide no evidence that computer-based narrative games offer a superior venue for academic learning under short time spans of under 2 hr. Findings contradict the discovery hypothesis that students learn better when they do hands-on activities in engaging scenarios during learning and the narrative hypothesis that students learn better when games have a strong narrative theme, although there is no evidence concerning longer periods of game play.

Gee, that “under two hours” point seems like an important limitation of the research, maybe one that should have been mentioned by Clark when claiming that games have no value and “don’t teach.”

It’s also possible that there are flaws in the research.

  • The research says that the games were “well designed,” but maybe they actually weren’t. Maybe they were “well designed” by the standards of traditional courses, but not by the standards of games. Without seeing the full article, I can’t tell.
  • The learners did worse at “retention,” but honestly, I wouldn’t expect a narrative game to be all that effective at helping people memorize content. If retention was the goal, a narrative discovery style game probably was the wrong approach, which brings us back to the previous point about whether the course was well designed for the goals.
  • One of the benefits of games for learning is application and behavior change, something this research didn’t measure. I’m not terribly surprised that a game with hands-on practice didn’t help people simply recall information that well. I would have liked to see some measure of how well the learners could apply the concepts. But, as is also typical of Clark’s work, the focus is on whether people recall content, not whether they can apply it. This, strictly speaking, is a limitation of the research and not a flaw, but it is something we should consider when looking at how we apply this research to our work.

I think there’s a case to be made that the games themselves weren’t actually “well designed” as claimed. They didn’t allow for real practice, just a different format for receiving content. In the discussion on this post in the eLearning Guild’s LinkedIn group, Cathy Moore made this observation:

I don’t have access to the full study cited in Ruth’s article, but based on the description of the games in the abstract, (1) they don’t simulate a realistic situation that’s relevant to the learners and (2) they teach academic info that learners aren’t expected to apply in real life. The material was tested on college students in an academic setting, not adults on the job.

By requiring learners to explore (or slog though, in my opinion!) an irrelevant treasure hunt, you’re adding cognitive load or at the least distracting the brain from the content. It seems likely to me that putting the material in a more relevant context, such as using your knowledge of pathogens to protect patients in a hospital, would have changed the results of the study.

As Ruth herself says in the comments to the article, “I think it’s about designing a simulation (which I don’t equate directly to games) in a manner that allows learners to practice job-relevant skills.” Neither of those games let students practice job- or life-relevant skills. They were entertaining and distracting ways of presenting information for a test.

Another limitation is that this research can’t address the question of engagement and completion rates. In the real world, getting people to complete online learning is often a challenge. If your traditional text-based click next slide presentation course has a less than 20% completion rate, then a game that is engaging enough to make people want to finish and gets completion rates above 90% is a big improvement—even if that game technically produced lower retention rates in a controlled lab environment. Learning doesn’t always have to be drudgery, although sometimes we equate “worthwhile” with “unpleasant.” There is value in making it interesting enough to keep people’s attention, and maybe even an enjoyable experience.

In the previously mentioned discussion, Tahiya Marome made this point:

For the brain, play is learning and learning is play. That traditional educational structures have sucked that dry and replaced it with a grim Puritanical work is learning and learning is work structure doesn’t mean we have to leave it that way. It may take us a while to figure out exactly how, but we can make educating oneself playful and a great, life long game again. We can. Our brains are wired for it.

Clark has some legitimate points about the definition of games being fuzzy and that the design of the game should match the learning outcomes. For example, I agree with her that adding a timer to critical thinking tasks can be counterproductive. Adding a timer to skill practice for skills that really do need to be timed is good practice though. Think of help desk agents who are evaluated both on the quality of their service and how quickly they can solve problems; timed practice matches the learning outcomes.

If Clark is going to make the claim that “games don’t teach,” she needs to address all the research that contradicts her point. She makes this claim without even mentioning any of the other research.; she just pretends nothing else exists beyond the one study cited. That is, frankly, an extraordinary claim, and extraordinary claims require extraordinary evidence. One study doesn’t discount the dozens of successful examples out there. It’s bad use of research to treat any individual study as applying in all situations, regardless of the limitations of the study. What we need to look at is the trends across the bulk of research, not a single data point. There are definitely bad games out there, and games aren’t the solution in every situation, but that doesn’t mean games shouldn’t be one tool in our toolbox like Clark claims.

Here’s a cursory review of a few examples of successful games for learning. This is by no means a comprehensive review, but this would be a good place for Clark to start refuting evidence if she wants to dissuade people from using games. Again, the point is not to look at any single study as being the end of the discussion, but to look at the overall findings and the types of strategies that have repeatedly been shown to work.

  • Immersive games beats classroom in maths, summarized by Donald Clark
  • Via Karl Kapp, from a past presentation:
    • “Trainees learn more from simulations games that actively engage trainees in learning rather than passively conveying the instructional material.”
    • “Trainees participating in simulation game learning experiences have higher declarative knowledge, procedural knowledge and retention of training material than those trainees participating in more traditional learning experiences.”
  • Eduweb has a collection of research related to games for learning. Here’s a highlight from the findings of one paper: “Summative evaluation of our WolfQuest wildlife simulation game finds that players report knowledge gain, stronger emotional attachment to wolves, and significant behavioral outcomes, with large percentages of players following their game sessions with other wolf-related activities, including such further explorations of wolves on the internet, in books and on television.”
  • Kurt Squire has done extensive research in games for learning. Clark basically needs to disprove all of his work to support her claim.
  • Clark Aldrich has created a number of successful games and simulations, such as Virtual Leader. “Using practiceware significantly increased retention and application, not just awareness of learned content.”
  • James Paul Gee has published a number of articles on games and learning.
  • Mark Wagner’s dissertation on MMORPGs in education found that “MMORPGs may help students develop difficult to teach 21st Century skills and may be used to support student reflection.”
  • The Educational Games Research blog features exactly what you would think it does based on the title.

Thanks to Cathy and Tahiya for giving me permission to quote them here!

I’d love to hear if any of you out there have designed games for learning and found them to be effective or not. I’ll have more to say about this topic next week in my post for the blog book tour for the Gamification of Learning and Instruction.

20 thoughts on “Ruth Clark Claims “Games Don’t Teach”

  1. I had similar thoughts when I read the article and considered leaving a comment but didn’t have the time. You’ve comprehensively addressed my initial apprehensions and more.

    The comparison made in Ruth’s article highlights a comparison more akin to: Complex media assemblies masquerading as a game don’t outperform less complex media conveyance.

    It’s an interesting discussion and I think it does highlight the fact that pseudo-games or games as bolt-on components don’t pass the litmus test.

    Well designed ANYTHING will provide benefits. The games you reference and the research you cite imply that well designed games leverage the science of psychology (not simply polishing up visual concepts) and either by design or by side-effect certainly do provide learning benefits.

    1. Thanks, Steve. I think the research itself does support a conclusion that discovery narrative style games aren’t effective in the situation they measured (with college students who were paid or received course credit for participation). That’s a very specific kind of conclusion though.

      Clark even hints at that when she says that “it is just as meaningless to generalize the learning value of games as to make claims about the learning value of graphics or other common training methods.” She’s right; we can’t make an overarching conclusion that games are either always good or always bad. Unfortunately, she didn’t take her own advice, and she made this sweeping generalization.

      As you said, well-designed learning experiences that are matched to the audience and learning outcomes is what we need. For that, we can’t just look superficially at research or pretend that any single study is the end of the conversation.

  2. While I am just starting in the field and have not had the chance to design games for learning, I think that games that are carefully designed can be as effective in education as any presentations or multimedia we often use as long as they capture the student’s attention and allow the student to create associations with existing ideas so they can be easily stored and retrieved in long-term memory. What Clark doesn’t seem to understand is that these games are simply tools that can help in the teaching and learning process. Students learn in different ways and not all techniques will work for all of them. I think we should take any available tools so that our students can learn one way or another.

    1. I think Ruth understands. This was my initial reaction to the article as well. I took another look at the piece. The specific case used in the comparison points to a great point. When something simpler will do the job as well or better than something more complicated, pick the simpler thing.

      I don’t think Ruth meant to express that “games don’t teach” as the title seems to say. She knows (she’s been doing this quite a long time and is pretty esteemed in the area of media for learning) that media can help to facilitate learning and support a job of teaching a concept, idea or skill. Games are a form of media. I think what she’s pointing out is that games are an expensive media component. Sometimes it’s not worth the extra effort. Effort to outcome doesn’t balance. There’s also a cognitive cost to games that needs to be considered. I think both of these factors are things that Ruth points out in her article.

      In my experience, the best instructional experiences in games have been designed by those that don’t have instructional design conditioning. I’m not sure that points to a specific detriment created by ISD training or just a lack of game design skill. The worst examples of games I’ve seen have been led by instructional design folk. Just my experience. YMMV.

      1. Rafael, I agree that we have lots of tools available. Different strategies are better in different situations. Steve, there is some of that in Clark’s article, but her overall conclusion is still pretty strongly anti-gaming.

        She says, “At this stage, I recommend games to implement drill and practice exercises for tasks that require immediate and accurate responses.”

        Clark doesn’t actually provide any research to back up that recommendation, although those citations are available.

        I do agree with the final sentence though: “If you are determined to gamify, I recommend testing a prototype version to evaluate its effectiveness and efficiency compared to a more traditional approach.”

        Prototypes are definitely a good strategy, and that is a sound recommendation.

        Overall though, I don’t think you title your post “Why Games Don’t Teach” if your goal isn’t to dissuade people from using games. You don’t have that goal unless you don’t think gaming is a valid strategy.

  3. Hey Christie — I completely agree. I’m trying to be charitable about the article title — I understand the power of provocative titles, and have used that myself, but I think if you do that, you have an obligation to clarify in the article.

    When the title is patently wrong and easily falsifiable, you have to own that in the article. Additionally, overgeneralizing from a single research study is a problem – it’s valuable to make people aware of it, but it is a single data point, and it could be a badly designed learning game.

    Also, thanks for the great round up of games+learning links!

  4. You make some excellent points in your post, Christy. For your own edification or further research, I thought I would share the following resources with you.

    The International Game Developers Association has two SIGs (Special Interest Groups) that touch on games and learning: the Serious Games SIG and the Learning, Education and Games SIG. While much of their discussions are limited to members, many of the individual members are engaged in research in these areas and I think you can look up members without logging in as a member. The site is: http://www.igda.org/

    Additionally there are at least two organizations dedicated to research on video games in general, and they probably have research on educational games.

    Center for Computer Games Research: http://game.itu.dk/index.php/About

    Digital Games Research Association:
    http://www.digra.org/

    Also, a Google search on “game-based learning” brought up several articles on the topic that might be of interest to you.

    1. Excellent resources! Most of the research I have looked at is from within the education world, so it’s nice to have a glimpse of the work being done within the broader gaming industry too. Thanks for sharing!

  5. Pingback: Games Teach!
  6. I like your moxy to acknowledge valid points in someone’s point, even when a writer is clearly taking shots at you. Anyway, aren’t ad hominem arguments considered fallacies of rhetoric, therefore making the speaker look pretty lame?

    I’ve shown my colleagues Clark’s post and your rebuttal (I’m an instructional designer/SME at a educational publishing firm, where most of what we make are math and language arts games for eLearning), and I’m still scratching my head wondering what Clark was up to. Why would she say such outrageous things about games, except maybe to stir up a hornet’s nest? Is she just trying to say that the delivery medium must be justified by the content, and that games are not a one-size-fits-all?

    Although Clark’s “Graphics for Learning” book is very well-respected at my workplace, this definitely does not make her a god, nor even necessarily right.

    It is strange that she would use “Games Don’t Teach” as a title, just as an attention getter.

    Thanks for all your posts. As a budding instructional designer, you’ve given me plenty of places to look for resources.

    1. Thanks for the compliment, Eric. (For any other readers wondering about the context, see my bookmarks post on Guy Wallace’s Reset post, where he attacked me for this post.)

      I admit I’m not a regular reader of Wallace’s blog, but I saw several posts where he made similar attacks on (usually unnamed) authors. Wallace is clearly focused on people’s credentials rather than their arguments. I tend to be on the opposite end of the spectrum; I want to respond to the arguments people make rather than who the person is. People with great credentials (like Ruth Clark) sometimes make mistakes; people who are unknowns in the field with no formal training sometimes have brilliant ideas. Even someone who is wrong some of the time can be right some of the time too. Wallace certainly does show a logical fallacy in his ad hominem attacks, but that doesn’t mean that everything he says is wrong. I would rather model the behavior of responding to arguments rather than people and show Wallace a different way than stoop to his level.

      As for Clark’s title, I’ve seen some other people raise the question of whether that title was really written by her or by an editor at ASTD. I don’t know if that’s the case, but it’s possible. The title certainly was chosen with the goal of generating controversy (to some extent, so was mine), whether it was really Clark’s words or those of an editor. Even if we ignore the title and just look at her concluding paragraph though, Clark still makes a seriously overgeneralized argument based on a single, limited study. If she was really trying to primarily make the point “that the delivery medium must be justified by the content, and that games are not a one-size-fits-all” as you said, I don’t think her final conclusion about using them only for drill and practice makes that point strongly.

  7. I actually read the initial paper today. I found this page because I was actually looking for the site where I had it stored.

    I think you hit the nail on the head with the research methodology being suspect (and then Ruth Clark extrapolating from that). Among the elements that you didn’t include: the authors of the article themselves describe the games as “tedious,” which directly contradicts their statements about the games being well-designed. Second, reading through the references cited, it’s clear that these people really don’t like Constructivism, or Experiential Learning, or Problem-Based Learning. I can’t help but wonder if they have a research bias (not that pro-game educational researchers don’t have the same problem). Finally, I’m doing a meta-review of the literature on the (admittedly limited) quantitative research about the efficacy of games in education and I can’t help but notice that of the 114 research articles I read, there are only 4 that warn of the dangers of games… and three of them are in the bibliography of Clark’s article. More tellingly, all four cite each other. I don’t want to drop my pretense of academic poise, but that kind of recursive citation is colloquially referred to as a “circle jerk.”

    1. Thanks for the information on the original article. I hate to admit it, but none of what you said surprises me (except perhaps their own admission that the game is “tedious”).

      Both sides of the issue can be swayed by what they want to find. Confirmation bias is hard to avoid even when you know what it is and how to recognize it.

      1. A little shameless promotion: I’ll be presenting my meta-research in a two hour seminar entitled “Convincing Administrators And Parents That Games Belong In Education” next wednesday at the Gen Con Educator’s Trade Day. Specifically, I’ll be picking apart arguments like Ruth Clark’s and explaining how to put minds at ease when trying something new and innovative in a classroom.

        Then I will be gloriously indulging in four straight days of games (free of educational context).

        1. Best of luck with your presentation. The change management aspect of games (and all innovation) in education is critical.

          After that, enjoy your well-earned time off!

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s