“Benny’s Rules” Still Rule
Part 4 of a series 

MARCH 1, 2012. In designing a video game to help students learn mathematics, it’s important not to over-estimate the capabilities of the medium. Breathless articles about the imminent arrival of HAL-like artificial intelligence notwithstanding, the day is not yet here (and in my view won’t be for a long time, if indeed at all) when we can take the human teacher out of the loop. If you really want to develop video games that contribute in a significant way to mathematics education, you should view them as supplementary educational materials or tools to be used by teachers.

A major problem with video games, or more generally any mechanized educational delivery system, is that the system has no way of knowing what the player, or student, is learning. That a player who moves up a level in a video game has learned something is clear. Video games are all about learning. But all you can reliably conclude from a player’s leveling up is that she or he has leveled up. It could have been happenstance.

Okay, but what if the player keeps leveling up? Surely that is not just chance? Possibly not; indeed, given good game design, probably not. The player must have learned something. But what? It might be precisely what that game activity was designed to teach. But it could be something quite different.

This is not a problem unique to video games, or to educational technology in general. It’s a fundamental problem about teaching and learning.

Take a look at the following video from the well-known educational consultant Marilyn Burns:

If you are like me the first time I saw this video, when you heard Cena’s answer in the class you concluded that she understood place value representation. She certainly gave the right answer. Moreover, to those of us who do understand place-value, her verbally articulated reasoning indicated she had conceptual understanding. But she had nothing of the kind, as the subsequent interview made clear.

And therein lies the problem. The human brain is a remarkable pattern-recognizing device. It will even discern a pattern – usually many patterns – in a random display of dots on a screen, where by definition there is no pattern. But is it the pattern the brain recognizes “right” pattern? Cena clearly recognized a pattern, and it yielded the “right answer.” But was it place-value? Perhaps some aspect, but we have no way of knowing

Of course, video games are highly interactive and ongoing. Surely, with video game learning a false understanding will eventually become apparent. Eventually the player will demonstrate that something has gone wrong. Right?

Unfortunately not, as was discovered in 1973, albeit not in the context of a video game but something with similar features. In what rapidly became one of the most famous and heavily studied papers in the mathematics education research literature, Stanley Erlwanger exposed the crippling limitations of what at the time was thought to be a major step forward in mathematics education: Individually Prescribed Instruction (IPI).

Though not a video game, nor indeed delivered by any technology beyond printed sheets of paper, IPI was very similar to an educational video game, in that it presented students with a series of mathematical problems that were selected and delivered at a rate thought to be ideally suited to the individual student, leading the student forward in the same way good level design does in a video game. Without a doubt, if IPI has problems, so too will video games. And as Erlwanger’s paper “Benny’s Conception of Rules and answers in IPI Mathematics” showed, IPI had problems. Big problems. How big? It didn’t work.

The subject of Erlwanger’s study was a twelve-year-old boy called Benny, chosen because he was doing particularly well on the program, moving rapidly from level to level, scoring highly at each stage. As Erlwanger states in his paper, Benny’s teacher, who was administering the program for Benny, felt sure that his pupil could not have progressed so far without having a good understanding of previous work.

Erlwanger’s research methodology was essentially the same as the approach Marilyn Burns used. He interviewed Benny to see what the boy understood. And when he did, a large can of worms spilled out. Though he got high scores on all the question sheets, Benny had almost no understanding of any mathematics, and a totally warped view of what mathematics is, to boot.

Being bright, Benny had quickly worked out a strategy for tacking the IPI question sheets. His strategy was based in part on pattern recognition, and in part on developing a theory about how the game was constructed – yes, he viewed it as a game! And he did what any smart kid would do, he figured out how to game the game.

What the designers of the IPI program had intended was that gaming the game required mastering the mathematics. Unfortunately, there is no way to prevent people, particularly smart ones, from coming up with alternative systems.

In Benny’s case, this involved developing a complete set of rules for adding, subtracting, multiplying and dividing fractions. Though his rules were symbolic manipulation procedures that made no sense mathematically, they enabled him to move through the sheets faster than everyone else in his cohort group, scoring 80% or better at each stage.

Whenever his rules yielded wrong answers, he simply adapted them to fit the new information he had acquired.

When asked by Erwanger, Benny was able to provide consistent, coherent explanations of his methods and why they worked. He was also very confident in his performance, and would stick to his explanations and would not alter his answers when pressured.

I won’t spend time here going through the details. You can read it all in Erlwanger’s paper, which is available here. Anyone who is about to embark on designing a video game for mathematics education should read that paper thoroughly. You need to know what you are up against. (The same dangers arise with gamification, and for the same reason.)

What I will do is say briefly what the fundamental issue is. The designer of the video game (just like the developers of the IPI worksheets) starts with an understanding of the mathematics to be learned, and creates a system to deliver it. The player, or student, does not yet know that mathematics, so they approach the system as what they see: a video game in our case or a series of quizzes in Benny’s. In both cases, the rewards come not from mastery of the underlying mathematics, but from successful completion of the challenge qua challenge. Indeed, with many educational video games, that’s the whole point: turn mathematics learning into a game!

In Benny’s case, not only was he successful in “playing the game”, in the process he developed an entire conception of mathematics as consisting of pointless questions that have a range of possible correct answers, one of which the test maker (in our case, read game developer) had decided, according to some secret but arbitrary set of rules, to declare as the “correct” one. Benny saw his task as to figure out the arbitrary rules the test-maker was using.

Only when you understand the nature of mathematics does Benny’s strategy seem crazy. Without such understanding, his approach is perfectly sensible. He does not know about math, but he already knows a lot about people and about playing games of different kinds. And when this particular game keeps telling him he is doing well, and making progress, he has no reason to change his basic assumptions.

Anyone who sets out to develop a math ed video game needs to have a strategy to avoid falling into the Benny Trap. Personally, I know of no way to do that with any hope of success other than conducting Marilyn Burns type player interviews throughout the development cycle. Fortunately, game developers are already used to doing lots of player testing. Mostly, they are checking for playability and engagement. With an educational video game, they need to augment those tests with interviews to see what is being learned.

That should at least ensure that the game will stand a chance of achieving the educational goal you want. The next issue to address is the circumstances under which the game will be played, and in particular the role of the (human) teacher.

To be continued …