How to design video games that support good math learning: Level 4

“Benny’s Rules” Still Rule
Part 4 of a series

MARCH 1, 2012. In designing a video game to help students learn mathematics, it’s important not to over-estimate the capabilities of the medium. Breathless articles about the imminent arrival of HAL-like artificial intelligence notwithstanding, the day is not yet here (and in my view won’t be for a long time, if indeed at all) when we can take the human teacher out of the loop. If you really want to develop video games that contribute in a significant way to mathematics education, you should view them as supplementary educational materials or tools to be used by teachers.

A major problem with video games, or more generally any mechanized educational delivery system, is that the system has no way of knowing what the player, or student, is learning. That a player who moves up a level in a video game has learned something is clear. Video games are all about learning. But all you can reliably conclude from a player’s leveling up is that she or he has leveled up. It could have been happenstance.

Okay, but what if the player keeps leveling up? Surely that is not just chance? Possibly not; indeed, given good game design, probably not. The player must have learned something. But what? It might be precisely what that game activity was designed to teach. But it could be something quite different.

This is not a problem unique to video games, or to educational technology in general. It’s a fundamental problem about teaching and learning.

Take a look at the following video from the well-known educational consultant Marilyn Burns:

If you are like me the first time I saw this video, when you heard Cena’s answer in the class you concluded that she understood place value representation. She certainly gave the right answer. Moreover, to those of us who do understand place-value, her verbally articulated reasoning indicated she had conceptual understanding. But she had nothing of the kind, as the subsequent interview made clear.

And therein lies the problem. The human brain is a remarkable pattern-recognizing device. It will even discern a pattern – usually many patterns – in a random display of dots on a screen, where by definition there is no pattern. But is it the pattern the brain recognizes “right” pattern? Cena clearly recognized a pattern, and it yielded the “right answer.” But was it place-value? Perhaps some aspect, but we have no way of knowing

Of course, video games are highly interactive and ongoing. Surely, with video game learning a false understanding will eventually become apparent. Eventually the player will demonstrate that something has gone wrong. Right?

Unfortunately not, as was discovered in 1973, albeit not in the context of a video game but something with similar features. In what rapidly became one of the most famous and heavily studied papers in the mathematics education research literature, Stanley Erlwanger exposed the crippling limitations of what at the time was thought to be a major step forward in mathematics education: Individually Prescribed Instruction (IPI).

Though not a video game, nor indeed delivered by any technology beyond printed sheets of paper, IPI was very similar to an educational video game, in that it presented students with a series of mathematical problems that were selected and delivered at a rate thought to be ideally suited to the individual student, leading the student forward in the same way good level design does in a video game. Without a doubt, if IPI has problems, so too will video games. And as Erlwanger’s paper “Benny’s Conception of Rules and answers in IPI Mathematics” showed, IPI had problems. Big problems. How big? It didn’t work.

The subject of Erlwanger’s study was a twelve-year-old boy called Benny, chosen because he was doing particularly well on the program, moving rapidly from level to level, scoring highly at each stage. As Erlwanger states in his paper, Benny’s teacher, who was administering the program for Benny, felt sure that his pupil could not have progressed so far without having a good understanding of previous work.

Erlwanger’s research methodology was essentially the same as the approach Marilyn Burns used. He interviewed Benny to see what the boy understood. And when he did, a large can of worms spilled out. Though he got high scores on all the question sheets, Benny had almost no understanding of any mathematics, and a totally warped view of what mathematics is, to boot.

Being bright, Benny had quickly worked out a strategy for tacking the IPI question sheets. His strategy was based in part on pattern recognition, and in part on developing a theory about how the game was constructed – yes, he viewed it as a game! And he did what any smart kid would do, he figured out how to game the game.

What the designers of the IPI program had intended was that gaming the game required mastering the mathematics. Unfortunately, there is no way to prevent people, particularly smart ones, from coming up with alternative systems.

In Benny’s case, this involved developing a complete set of rules for adding, subtracting, multiplying and dividing fractions. Though his rules were symbolic manipulation procedures that made no sense mathematically, they enabled him to move through the sheets faster than everyone else in his cohort group, scoring 80% or better at each stage.

Whenever his rules yielded wrong answers, he simply adapted them to fit the new information he had acquired.

When asked by Erwanger, Benny was able to provide consistent, coherent explanations of his methods and why they worked. He was also very confident in his performance, and would stick to his explanations and would not alter his answers when pressured.

I won’t spend time here going through the details. You can read it all in Erlwanger’s paper, which is available here. Anyone who is about to embark on designing a video game for mathematics education should read that paper thoroughly. You need to know what you are up against. (The same dangers arise with gamification, and for the same reason.)

What I will do is say briefly what the fundamental issue is. The designer of the video game (just like the developers of the IPI worksheets) starts with an understanding of the mathematics to be learned, and creates a system to deliver it. The player, or student, does not yet know that mathematics, so they approach the system as what they see: a video game in our case or a series of quizzes in Benny’s. In both cases, the rewards come not from mastery of the underlying mathematics, but from successful completion of the challenge qua challenge. Indeed, with many educational video games, that’s the whole point: turn mathematics learning into a game!

In Benny’s case, not only was he successful in “playing the game”, in the process he developed an entire conception of mathematics as consisting of pointless questions that have a range of possible correct answers, one of which the test maker (in our case, read game developer) had decided, according to some secret but arbitrary set of rules, to declare as the “correct” one. Benny saw his task as to figure out the arbitrary rules the test-maker was using.

Only when you understand the nature of mathematics does Benny’s strategy seem crazy. Without such understanding, his approach is perfectly sensible. He does not know about math, but he already knows a lot about people and about playing games of different kinds. And when this particular game keeps telling him he is doing well, and making progress, he has no reason to change his basic assumptions.

Anyone who sets out to develop a math ed video game needs to have a strategy to avoid falling into the Benny Trap. Personally, I know of no way to do that with any hope of success other than conducting Marilyn Burns type player interviews throughout the development cycle. Fortunately, game developers are already used to doing lots of player testing. Mostly, they are checking for playability and engagement. With an educational video game, they need to augment those tests with interviews to see what is being learned.

That should at least ensure that the game will stand a chance of achieving the educational goal you want. The next issue to address is the circumstances under which the game will be played, and in particular the role of the (human) teacher.

To be continued …

How to design video games that support good math learning: Level 3

The Symbol Barrier
Part 3 of a series

FEBRUARY 22, 2012. In my view, the most significant single benefit that video games offer to mathematics education is their capacity to overcome the biggest obstacle to practical mastery of middle school math: the symbol barrier. Yet to date, practically none of the now hundreds of math ed video games available have even begun to address it. In part, I suspect, because the developers of those games were probably not aware of the issue.

Chances are you have never heard of the symbol barrier either. Certainly not by that name, I agree. That term is mine, and I started using it only recently (when I realized that video games provided the key to overcome it). But the problem itself has been familiar to mathematics learning specialists for twenty years, and it created a considerable stir when it was first observed. The first main chapter of my recent book on mathematics education video games, after the opening chapter that sets the scene, is devoted to a fairly lengthy discussion of the issue.

To understand the symbol barrier, and appreciate how pervasive it is, you have to question the role symbolic expressions play in mathematics.

When a TV or movie director wants the audience to know that a particular character is a mathematician, somewhere in that character’s first scene you will see her or him writing symbols – on a piece of paper, on a blackboard, or, quite likely, on a window or a bathroom mirror. (Real mathematicians never do that, but it looks cool on the screen.) This character-establishing device is so effective because, as the director knows very well, people universally identify doing math with writing symbols, often obscure symbols.

Why do we make that automatic identification? Part of the explanation is that much of the time we spent in the school mathematics classroom was devoted to the development of correct symbolic manipulation skills, and symbol-filled books are the standard way to store and distribute mathematical knowledge. So we have gotten used to the fact that mathematics is presented to us by way of symbolic expressions.

But just how essential are those symbols? After all, until the invention of various kinds of recording devices, symbolic musical notation was the only way to store and distribute music, yet no one ever confuses music with a musical score.

Just as music is created and enjoyed within the mind, so to is mathematics created and pursued (and by many of us enjoyed) in the mind. At its heart, mathematics is a mental activity – a way of thinking. Not a natural way of thinking, to be sure; rather one that requires training to learn and concentration to achieve. But a way of thinking that over several millennia of human history has proved to be highly beneficial to life and society.

In both music and mathematics, the symbols are merely static representations on a flat surface of dynamic mental processes. Just as the trained musician can look at a musical score and hear the music come alive in her or his head, so too the trained mathematician can look at a page of symbolic mathematics and have that mathematics come alive in the mind.

So why is it that many people believe mathematics is symbolic manipulation? And if the answer is that it results from our classroom experiences, why is mathematics taught that way? I can answer that second question. We teach mathematics symbolically because, for many centuries, symbolic representation has been the most effective way to record mathematics and pass on mathematical knowledge to others.

Still, given the comparison with music, can’t we somehow manage to break free of that historical legacy?

Well, things are not quite so simple. Like all analogies, the comparison of mathematics with music, while helpful, only takes you so far. Although mathematical thinking is a mental activity, for the most part the human brain can do it only when supported by symbolic representations. In short, the symbolic representation seems far more crucial to doing mathematics than is musical notation for performing music. (We are all aware of successful musicians who cannot read or write a musical score.) In fact, much of mathematics – including all advanced mathematics – deals with symbolically defined, abstract entities. Without the symbols, there would be no entities to reason about.

The one exception, where the brain does not require the aid of symbolic representations (and where the comparison with music holds well) is what for several years now I have been calling “everyday mathematics.” This is the collection of mathematical concepts, operations, and procedures that are an essential part of everyday life skills for today’s world – the mathematical equivalent of the ability to read and write. (In contrast to the mathematics required for science, engineering, economics, advanced finance, and many parts of business, where fluency with symbolic expressions is essential.)

Roughly speaking, everyday mathematics comprises counting, arithmetic, proportional reasoning, numerical estimation, elementary geometry and trigonometry, elementary algebra, basic probability and statistics, logical thinking, algorithm use, problem formation (modeling), problem solving, and sound calculator use. (Yes, even elementary algebra belongs in that list. The symbols are not essential. For much of its roughly fifteen-hundred-year history, algebra was not written down symbolically, rather was recorded, described, and taught using ordinary language, with terms like “the unknown” where today we would write an “x”.)

True, people sometimes scribble symbols when they do everyday math in a real-life context. But for the most part, what they write down are the facts needed to start with, perhaps the intermediate results along the way and, if they get far enough, the final answer at the end. But the doing math part is primarily a thinking process – something that takes place primarily in your head. Even when people are asked to “show all their work,” the collection of symbolic expressions that they write down is not necessarily the same as the process that goes on in their heads when they do math correctly. In fact, people can become highly skilled at doing mental math and yet be hopeless at its symbolic representations.

It is with everyday mathematics that the symbol barrier emerges.

In the early 1990s, three researchers, Terezinha Nunes (then at the University of London, England, now at Oxford University), Analucia Dias Schliemann, and David William Carraher (both of the Federal University of Pernambuco in Recife, Brazil) embarked on an anthropological study in the street markets of Recife. With concealed tape recorders, they posed as ordinary market shoppers, seeking out stalls being staffed by young children between 8 and 14 years of age. At each stall, they presented the young stallholder with a transaction designed to test a particular arithmetical skill. The purpose of the research was to compare traditional instruction (which all the young market traders had received in school since the age of six) with learned practices in context. In many cases, they made purchases that presented the children with problems of considerable complexity.

What they found was that the children got the correct answer 98% of the time. “Obviously, these were not ordinary children,” you might imagine, but you’d be wrong. There was more to the study. Posing as shoppers and recording the transactions was only the first part. About a week after they had “tested” the children at their stalls, the three researchers went back to the subjects and asked each of them to take a pencil-and-paper test that included exactly the same arithmetic problems that had been presented to them in the context of purchases the week before, but expressed in the familiar classroom form, using symbols.

The investigators were careful to give this second test in as non-threatening a way as possible. It was administered in a one-on-one setting, either at the original location or in the subject’s home, and the questions were presented in written form and verbally. The subjects were provided with paper and pencil, and were asked to write their answer and whatever working they wished to put down. They were also asked to speak their reasoning aloud as they went along.

Although the children’s arithmetic had been close to flawless when they were at their market stalls – just over 98% correct despite doing the calculations in their heads and despite all of the potentially distracting noise and bustle of the street market – when presented with the same problems in the form of a straightforward symbolic arithmetic test, their average score plummeted to a staggeringly low 37%.

The children were absolute number wizards when they were at their market stalls, but virtual dunces when presented with the same arithmetic problems presented in a typical school format. The researchers were so impressed – and intrigued – by the children’s market stall performances that they gave it a special name: they called it street mathematics.

As you might imagine, when the three scholars published their findings (in the book Street Mathematics and School Mathematics, Cambridge University Press, Cambridge, UK, 1993), it created a considerable stir. Many other teams of researchers around the world carried out similar investigations, with target groups of adults as well as children, and obtained comparable results. When ordinary people are faced with doing everyday math regularly as part of their everyday lives, they rapidly achieve a high level of proficiency (typically hitting that 98% mark). Yet their performance drops to the 35 to 40% range when presented with the same problems in symbolic form.

It is simply not the case that ordinary people cannot do everyday math. Rather, they cannot do symbolic everyday math. In fact, for most people, it’s not accurate to say that the problems they are presented in paper-and-pencil format are “the same as” the ones they solve fluently in a real life setting. When you read the transcripts of the ways they solve the problems in the two settings, you realize that they are doing completely different things. (I present some of those transcripts in my book.) Only someone who has mastery of symbolic mathematics can recognize the problems encountered in the two contexts as being “the same.”

That, my friend, is the symbol barrier. It’s huge and it is pervasive. For the entire history of organized mathematics instruction, where we had no alternative to using static, symbolic expressions on flat surfaces in order to store and distribute mathematical knowledge, that barrier has prevented millions of people from becoming proficient in a cognitive skill-set of evident major importance in today’s world, on a par with the ability to read and write.

With video games, we can circumvent the barrier.

To be continued …

How to design video games that support good math learning: Level 2.

The educational goal
Part 2 of a series.

JANUARY 31, 2012. Anyone setting out to design a video game to help students learn mathematics should start out by reading – several times, from cover to cover – the current “bible” on K-12 mathematics education. It is called Adding it Up: Helping Children Learn Mathematics, and was published by the National Academies Press in 2001. The result of several years work by the National Research Council’s Mathematics Learning Study Committee, a blue-ribbon panel of experts assembled to carry out that crucial millennial task, this invaluable volume sets out to codify the mathematical knowledge and skills that are thought to be important in today’s society. As such, it provides the best single source currently available for guidelines on good mathematics instruction.

The report’s authors use the phrase mathematical proficiency to refer to the aggregate of mathematical knowledge, skills, developed abilities, habits of mind, and attitudes that are essential ingredients for life in the twenty-first century. They then break this aggregate down to what they describe as “five tightly interwoven” threads:

Conceptual understanding – the comprehension of mathematical concepts, operations, and relations

Procedural fluency – skill in carrying out arithmetical procedures accurately, efficiently, flexibly, and appropriately

Strategic competence – the ability to formulate, represent, and solve mathematical problems arising in real-world situations

Adaptive reasoning – the capacity for logical thought, reflection, explanation, and justification

Productive disposition – a habitual inclination to see mathematics as sensible, useful, and worthwhile, combined with a confidence in one’s own ability to master the material.

The authors stress that it is important not to view these five goals as a checklist to be dealt with one by one. Rather, they are different aspects of what should be an integrated whole. On page 116 of the report, they say [emphasis in the original, image reproduced with permission]:

The most important observation we make here, one stressed throughout this report, is that the five strands are interwoven and interdependent in the development of proficiency in mathematics. Mathematical proficiency is not a one-dimensional trait, and it cannot be achieved by focusing on just one or two of these strands. … [W]e argue that helping children acquire mathematical proficiency calls for instructional programs that address all its strands. As they go from pre-kindergarten to eighth grade, all students should become increasingly proficient in mathematics.

In my book, I describe in some detail how to incorporate these educational goals (actually, to be faithful to the NRC Committee’s recommendation, I should say “educational goal”, in the singular) into good game design for a video game that seeks to help children learn mathematics. In this post, I’ll simply distill from that discussion eight important things to avoid. Try using this list to evaluate any math ed video game on the market. Very few – and I mean VERY few – pass through this filter.

AVOID: Confusing mathematics (a way of thinking) with its (symbolic) representation on a static, flat surface. (cf. music and music notation.)
AVOID: Presenting the mathematical activities as separate from the game action and game mechanics.
AVOID: Relegating the mathematics to a secondary activity when it should be the main focus.
AVOID: Reinforcing the perception that math is an obstacle that gets in the way of doing more enjoyable things.
AVOID: Reinforcing the perception that math is an arbitrary hurdle to be overcome, or circumvented, in order to progress .
AVOID: Encouraging the student to try to answer quickly, without reflection.
AVOID: Reinforcing the belief that math is just a large bag of isolated facts and tricks.
AVOID: Reinforcing the perception that math is so intrinsically uninteresting it has to be sugar coated.

I’ll be referring to Adding It Up a lot in this series. I shall also discuss many things TO DO when designing a good video game that support good learning, not just what to avoid. As you might (and for sure should) realize, with two challenging goals, good game and good learning, designing a successful math ed video game is difficult. Very difficult. If you do not have an experienced and knowledgable mathematics education specialist on your team, you are not going to succeed. Period.

Game programmers who think that because they were good at basic math (and they have to have been to become successful programmers) they can design a video game that will provide good learning are deluding themselves.

It’s easy to underestimate the depth of expertise of professionals in areas other than our own. Let me stress this point from the perspective of a hypothetical math educator who knows how to program in html5 and decides to create the next Angry Birds.

S/he might well think, “I can write code that produces screen action like that.” Indeed s/he probably can; it’s not hard. But as any experienced game developer will attest, the coding is the easiest part. The huge success of Angry Birds is not an accident. It is a result of brilliant design on many levels. (See this article for an initial, eye-opening summary of some of what went in to making that success.) The expertise it took to be able to create that game was acquired over many years. The Helsinki, Finland based Rovio game studio built ten other games, picking up a ton of increased expertise and insights along the way, before they reached the design heights of Angry Birds.

To build a successful game, you have to understand, at a deep level, what constitutes a game, how and why people play games, what keeps them engaged, and how they interact with the different platforms on which the game will be played. That is a lot of deep knowledge. On its own, being able to code is not enough.

To build a game that supports good mathematics learning, requires a whole lot more. You have to understand, at a deep level, what mathematics is, how and why people learn and do mathematics, how to get and keep them engaged in their learning, and how to represent the mathematics on the different platforms on which the game will be played. That too is a lot of deep knowledge. On its own, being “good at math”, or at least the relevant math, is not enough.

If you are a game developer who happens to have both kinds of expertise, then go ahead and build a game on your own. But I have yet to meet such a person. For the rest of us, the answer is clear. You need a team, and that team must have all the expertise you will require to do a good job. If that team does not include, in particular, an experienced, knowledgable, math education specialist, then you are not a good engineer. You are an amateur.