“Benny’s Rules” Still Rule
Part 4 of a series
In designing a video game to help students learn mathematics, it’s important not to over-estimate the capabilities of the medium. Breathless articles about the imminent arrival of HAL-like artificial intelligence notwithstanding, the day is not yet here (and in my view won’t be for a long time, if indeed at all) when we can take the human teacher out of the loop. If you really want to develop video games that contribute in a significant way to mathematics education, you should view them as supplementary educational materials or tools to be used by teachers.
A major problem with video games, or more generally any mechanized educational delivery system, is that the system has no way of knowing what the player, or student, is learning. That a player who moves up a level in a video game has learned something is clear. Video games are all about learning. But all you can reliably conclude from a player’s leveling up is that she or he has leveled up. It could have been happenstance.
Okay, but what if the player keeps leveling up? Surely that is not just chance? Possibly not; indeed, given good game design, probably not. The player must have learned something. But what? It might be precisely what that game activity was designed to teach. But it could be something quite different.
This is not a problem unique to video games, or to educational technology in general. It’s a fundamental problem about teaching and learning.
Take a look at the following video from the well-known educational consultant Marilyn Burns:
If you are like me the first time I saw this video, when you heard Cena’s answer in the class you concluded that she understood place value representation. She certainly gave the right answer. Moreover, to those of us who do understand place-value, her verbally articulated reasoning indicated she had conceptual understanding. But she had nothing of the kind, as the subsequent interview made clear.
And therein lies the problem. The human brain is a remarkable pattern-recognizing device. It will even discern a pattern – usually many patterns – in a random display of dots on a screen, where by definition there is no pattern. But is it the pattern the brain recognizes “right” pattern? Cena clearly recognized a pattern, and it yielded the “right answer.” But was it place-value? Perhaps some aspect, but we have no way of knowing
Of course, video games are highly interactive and ongoing. Surely, with video game learning a false understanding will eventually become apparent. Eventually the player will demonstrate that something has gone wrong. Right?
Unfortunately not, as was discovered in 1973, albeit not in the context of a video game but something with similar features. In what rapidly became one of the most famous and heavily studied papers in the mathematics education research literature, Stanley Erlwanger exposed the crippling limitations of what at the time was thought to be a major step forward in mathematics education: Individually Prescribed Instruction (IPI).
Though not a video game, nor indeed delivered by any technology beyond printed sheets of paper, IPI was very similar to an educational video game, in that it presented students with a series of mathematical problems that were selected and delivered at a rate thought to be ideally suited to the individual student, leading the student forward in the same way good level design does in a video game. Without a doubt, if IPI has problems, so too will video games. And as Erlwanger’s paper “Benny’s Conception of Rules and answers in IPI Mathematics” showed, IPI had problems. Big problems. How big? It didn’t work.
The subject of Erlwanger’s study was a twelve-year-old boy called Benny, chosen because he was doing particularly well on the program, moving rapidly from level to level, scoring highly at each stage. As Erlwanger states in his paper, Benny’s teacher, who was administering the program for Benny, felt sure that his pupil could not have progressed so far without having a good understanding of previous work.
Erlwanger’s research methodology was essentially the same as the approach Marilyn Burns used. He interviewed Benny to see what the boy understood. And when he did, a large can of worms spilled out. Though he got high scores on all the question sheets, Benny had almost no understanding of any mathematics, and a totally warped view of what mathematics is, to boot.
Being bright, Benny had quickly worked out a strategy for tacking the IPI question sheets. His strategy was based in part on pattern recognition, and in part on developing a theory about how the game was constructed – yes, he viewed it as a game! And he did what any smart kid would do, he figured out how to game the game.
What the designers of the IPI program had intended was that gaming the game required mastering the mathematics. Unfortunately, there is no way to prevent people, particularly smart ones, from coming up with alternative systems.
In Benny’s case, this involved developing a complete set of rules for adding, subtracting, multiplying and dividing fractions. Though his rules were symbolic manipulation procedures that made no sense mathematically, they enabled him to move through the sheets faster than everyone else in his cohort group, scoring 80% or better at each stage.
Whenever his rules yielded wrong answers, he simply adapted them to fit the new information he had acquired.
When asked by Erwanger, Benny was able to provide consistent, coherent explanations of his methods and why they worked. He was also very confident in his performance, and would stick to his explanations and would not alter his answers when pressured.
I won’t spend time here going through the details. You can read it all in Erlwanger’s paper, which is available here. Anyone who is about to embark on designing a video game for mathematics education should read that paper thoroughly. You need to know what you are up against. (The same dangers arise with gamification, and for the same reason.)
What I will do is say briefly what the fundamental issue is. The designer of the video game (just like the developers of the IPI worksheets) starts with an understanding of the mathematics to be learned, and creates a system to deliver it. The player, or student, does not yet know that mathematics, so they approach the system as what they see: a video game in our case or a series of quizzes in Benny’s. In both cases, the rewards come not from mastery of the underlying mathematics, but from successful completion of the challenge qua challenge. Indeed, with many educational video games, that’s the whole point: turn mathematics learning into a game!
In Benny’s case, not only was he successful in “playing the game”, in the process he developed an entire conception of mathematics as consisting of pointless questions that have a range of possible correct answers, one of which the test maker (in our case, read game developer) had decided, according to some secret but arbitrary set of rules, to declare as the “correct” one. Benny saw his task as to figure out the arbitrary rules the test-maker was using.
Only when you understand the nature of mathematics does Benny’s strategy seem crazy. Without such understanding, his approach is perfectly sensible. He does not know about math, but he already knows a lot about people and about playing games of different kinds. And when this particular game keeps telling him he is doing well, and making progress, he has no reason to change his basic assumptions.
Anyone who sets out to develop a math ed video game needs to have a strategy to avoid falling into the Benny Trap. Personally, I know of no way to do that with any hope of success other than conducting Marilyn Burns type player interviews throughout the development cycle. Fortunately, game developers are already used to doing lots of player testing. Mostly, they are checking for playability and engagement. With an educational video game, they need to augment those tests with interviews to see what is being learned.
That should at least ensure that the game will stand a chance of achieving the educational goal you want. The next issue to address is the circumstances under which the game will be played, and in particular the role of the (human) teacher.
To be continued …
profkeithdevlin – THE BLOG 
Regarding the Cena video, isn’t it the case that we as viewers can only recognize her mistakes because we have outgrown them? Even mathphobes watching the videos (from ages 13 onward) will recognize the mistakes Cena has made. Given that most of these people were educated under the traditional rote math system, how is this possible? Clearly, it isn’t as if in our later years, a more conceptual teacher taught us these things. What’s going on with Cena’s mistakes is probably a failure to tailor instruction at her Piagetian level of cognition. But as she ages, she clearly outgrows this. I’m curious to learn more about this..
I agree, of course, that conceptual understanding is lacking generally in mathematics education. Can you exemplify similar misunderstandings at higher levels of math that stay with the learner permanently?
Andrew: Thanks for writing.
Two examples of failure to properly master elementary concepts that endure into adulthood come to mind from personal experience.
One, as a university professor, I often found that students in advanced calculus were hampered by not having understood addition of fractions, both in terms of the mechanics and the rationale behind those mechanics. It causes problems in coming to terms with the definition of the derivative.
The other is one I’ve commented on a lot, namely the failure to understand multiplication. Admittedly multiplication is a very complex operation, though the fact that many adults do not even recognize that is worrying.
I have had exchanges with graduate students and occasionally professors who do not see multiplication as fundamentally different from addition. The troubles me because it is after all one of the two fundamental operations on numbers! So some conceptual misunderstandings or failures to fully grasp a concept do endure.
So while I would agree with you that in many (most?) cases, most of us manage to get most concepts right in the end, it is not always the case. And for the concepts we do eventually master, how many of them did we “get” because of the intervention of a good teacher?
Benny did not correct his massive misunderstandings despite many years of working on IPI (without teacher intervention).
In terms of significantly higher math, large numbers of people, including many professors of mathematics, fail to fully understand the Recursion Principle. I wrote about this in my MAA column last year: http://devlinsangle.blogspot.com/2011/11/how-multiplication-is-really-defined-in.html. (The focus is on recursion; multiplication is just a convenient example.)
There are others. I myself carried a major misunderstanding well into my research career, and it was only pointed out to me when a referee was evaluating a research paper I’d written. (I did manage to rescue the main proof.)
And what about all the people who submitted false proofs of Fermat’s Last Theorem that assumed unique factorization?
The trouble is, it usually takes someone else to point our our misconceptions.
KD
Thanks for this post. I like the connection to Benny. Totally appropriate here.
I’ve worked on designing some video games, and in my opinion it is deceptively difficult to incorporate actual mathematics. My team had several conversations with game programmers about the difference between players interacting with a demonstration of math concepts and players using math concepts in a meaningful way. Having been a middle school math teacher and a teacher coach, I’ve seen many lessons in which manipulatives were used similarly: to demonstrate math rather than being used to engage with concepts. It’s an interesting challenge to try to embed math concepts in a game without skewing toward a mere demonstration for which Cena, Benny, and their classmates think the game is about trying to guess the rules.
Belinda: I definitely agree as to the difficulty of embedding tasks into video games that require real mathematical thinking. Mind you, it’s not easy getting them into the classroom either. Thanks for your comment.
Hi Keith,
For me the lesson is that there are many more components to knowledge that us, who already have the knowledge, tend to realise.
To us, we know “10” in terms of counting out ten items. We also know “10” in terms of what it means to subtract ten items from a larger quantity of items. And we know that the “1” in “18” is the same thing.
We can use our knowledge so fluidly that it just seems like it’s the one bit of knowledge about “10” in all three contexts. The Cena video shows us that it isn’t. At one point we had to learn these separate things, but we have no memory of that experience (and we no doubt weren’t consciously aware of it at the time).
To me one of the most important goals of education is to understand what these underlying components of knowledge are.
If we know what they are we can test for them. And hopefully if we know what they are we can write mechanised tests for them. That way we could have mechanised ways of knowing what the player, or student, is learning.
Absolutely. The whole history of human activity has involved greater understanding and mastery that has enabled us to make routine and often mechanize what was once regarded as difficult. A superb illustration of that is the development of probability theory in the 17th century, a story I tell in my book “The Unfinished Game”. Of course, history also tells us that this process is never ending. We just push out the beach a bit; there is still an ocean out there. If we ever fool ourselves into thinking we have nailed education, leaving no role (and a key one) for teachers in education, then as a society we are doomed.
So here’s a couple of questions in relation to this issue.
1. It is assumed that we want to design a video game to teach specific math concepts. What if instead of teaching specific curricula mathematics, we wanted students to learn mathematical habits of mind?
2. Is it possible to create an environment which rewards mathematical thinking, rather than necessarily the product of that thinking? It seems to me obvious that if we reward the product rather than the process that we then necessarily will run into people who use a different process… perhaps even a completely incorrect process.
3. What would happen if we included a social aspect to these games (or frameworks for learning) so that instead of trying to learn in isolation, we have to be able to explain our reasoning to other people, during which (hopefully) our misconceptions become addressed?
4. How does this information apply to the Khan Academy?
David, As it happens I co-founded a small mobile games company to produce games that try to do exactly what you suggest. (This goal is also why my recent MOOC focused on mathematical thinking, rather than a set of curricular points.) We secured an initial round of angel investment a few months ago, and our first game should be released some time early next year. And we teamed up with an existing mobile game company with some successful social games to their credit in order to incorporate the social aspect and facilities/encourage group activity (and hence group learning). I doubt we’ll get everything right with version 1.0, but we will be gathering and analyzing the player data from day 1 onwards, and that will let us iterate and improve as we learn from our players. We also have some stellar math ed researchers lined up to help us with the learning side. The process-used issue is a huge one for any automated learning environment. The only solution I know of is to involve teachers in the loop. We have ideas about that, but implementing (and then testing) them will have to wait until our game (in fact, more than one) is out.
I’ll pass on your Khan Academy question, as any remark about KA brings out the trolls, and then I’ll have to close comments.
I’m looking forward to checking out the game when it comes out. This series of posts from you has been really interesting, and it has helped clarify some of my thinking in this area.
You’ve not addressed the social aspect. Perhaps your game could include what many games include – some sort of online discussion space for player? This way when questions about process do come up, the players (learners) have somewhere to go to address them, and perhaps some of the misconceptions about process can be addressed through peer to peer interactions.
The game itself is the stand-alone iPhone/iPad app (other versions will follow). Social media and multiplayer aspects will be provided on the website that supports the game. The intention is to have a whole series of mobile games that connect to the same website.