The beginnings of Education Science?

[This is a greatly expanded version of the essay published in Edge as my response to the Annual Edge Question for 2016.]

JANUARY 18, 2016. If you google the term “education science”, the search engine will return almost 2 billion hits. Yet even those of us who might be regarded as “in the field” will admit that there is no “there” there—nothing that could legitimately be called a science.

The education field is much like medicine was in the Nineteenth Century, a human practice, guided by intuition, experience, and occasionally inspiration. It took the development of modern biology and biochemistry in the early part of the Twentieth Century to provide the solid underpinnings of today’s science of medicine.

To be sure, since the second half of Twentieth Century, a great deal of work has been done on pedagogy, and much has been learned—though depressingly little has found its way into the classroom. But even the most ardent participant in that work would admit that the field could not be called a science, alongside, say, chemistry, physics, or medical science.

But that may be about to change. To me—a mathematician who became interested in mathematics education in the second half of my career—it seems that we may at last be seeing the emergence of a genuine science of learning. Given the huge significance of education in human society, if that is the case, then this will represent a major advance for Humanity.

At the risk—nay certainty—of raising the ire of many researchers, I should start out by observing that I am not basing my assessment on the rapid growth in popularity of educational neuroscience. You know, the kind of study where a subject is slid into an fMRI machine and asked to solve math puzzles. Those studies are valuable, and are undoubtedly science, but at the present stage, at best they provide some very tentative clues about how people learn, and little specific in terms of how to help people learn. (A good analogy would be trying to diagnose an engine fault in a car by moving a thermometer over the hood.)

Yes, I follow educational neuroscience research (mostly at Internet distance), and am often in admiration of the ingenuity of the researchers, but I don’t see it as even close to providing a solid basis for education the way, say, the modern theory of genetics advanced medical practice. One day? Maybe. But not yet.

Rather, I am encouraged in thinking we are seeing the emergence of a science of learning by the possibilities Internet technology brings to the familiar, experimental cognitive science approach.

The problem that has traditionally beset learning research has been its huge dependence on the individual teacher, which makes it near impossible to run the kinds of large scale, control group, intervention studies that are par-for-the-course in medicine. (No, I am not about to argue that computers will replace teachers! On the contrary, I am firmly of the opinion that teaching is inherently and inescapably a human–human endeavor.)

The problem raised by that inescapable centrality of the human teacher is that classroom studies invariably end up as studies of the teacher as much as of the students.

In fact, it is even worse. What those studies frequently measure is as much, if not more, the effect of the home environment of the students than what goes on in the classroom.

For instance, news articles often cite the large number of successful people who as children attended a Montessori school, a figure hugely disproportionate to the relatively small number of such schools. Now, it may well be the case (I think it is) that the Montessori educational principles are good, but it’s also the case that such schools are magnets for passionate, dedicated teachers and the pupils that attend them do so because they have parents who go out of their way to enroll their offspring in such a school, and already raise their children in a learning-rich home environment.

Internet technology offers an opportunity to carry out medical-research-like, large scale control group studies of classroom learning that can significantly mitigate the “teacher effect” and “home effect”, allowing useful studies of different educational techniques to be carried out. Provided you collect the right data, Big Data techniques can detect patterns that cut across the wide range of teacher–teacher and family–family variation, allowing useful educational conclusions to be drawn.

An important factor is that a sufficiently significant part of the actual learning is done in a digital environment, where every action can be captured.

This is not easily achieved. The vast majority of educational software products operate around the edges of learning: providing the learner with information; asking questions and capturing their answers (in a machine-actionable, multiple-choice format); and handling course logistics with a learning management system.

What is missing is any insight into what is actually going on in the student’s mind—something that can be very different from what the evidence shows, as was dramatically illustrated for mathematics learning several decades ago by a study now famously referred to as “Benny’s Rules”, where a child who had aced a whole progressive battery of programmed learning cycles was found (by a lengthy, human–human working session) to have constructed an elaborate internal, rule-based “mathematics” that enabled him to pass all the tests with flying honors, but was extremely brittle and bore no relation to actual mathematics.

But real-time, interactive software allows for much more than we have seen flooding out of tech hotbeds such as Silicon Valley.

To date, the more effective uses of interactive technology from the viewpoint of running large-scale, comparative learning studies, have been by way of learning video games—so-called game-based learning. But it remains an open question how significant is the game element in terms of learning outcomes. My bet, based on a few small studies, is on it being an important factor, but this is a question that can and will be answered by solid, scientific studies.

And there have been such studies. But it would be easy to have missed them.

With large numbers of technology companies, (as well as book publishers and occasionally media moguls) vying for education customers—a trend driven almost exclusively by expertise looking for new markets (hammers seeking new nails)—and making often wildly extravagant claims in the process, the small number of studies in the past few years that have shown some remarkable results have not garnered much media attention.

That is not entirely the fault of the news media. With results still largely tentative, and for the most part not fully explained, those involved in those studies (I am one of them) have been reluctant to go out on a limb with bold statements.

That reluctance is heightened by the depressing reality that the vast majority of so-called “learning games” are of a very poor quality and offer little or no real learning, relying instead on bold marketing claims. The last thing serious researchers want to do is find their claims dismissed as yet more vacuous hype.

To cut to the chase, in the case of elementary through middle school mathematics learning (which is the research I am familiar with), what has been discovered, by a number of teams, is that digital learning interventions of as little as ten minutes a day, for three to five days a week, over a course of as little as one month, can result in significant learning gains when measured by a standardized test—with improvements of as much as 16% in some key thinking skills. (I list some sources at the end.)

That may sound like an educational magic pill. It almost certainly is not. I believe it’s an early sign that we know even less about learning than we thought we did.

For one thing, part of what is going on is that many earlier studies measured the wrong things—knowledge rather than thinking ability. The learning gains found in the studies I am referring to are not in the area of knowledge acquired or algorithmic procedures mastered, rather in high-level problem solving ability. (So the standardized tests used cannot be multiple-choice; they require human grading or game-based assessment techniques.)

What is exciting about these findings, is that in today’s information- and computation- rich environment, those very human problem-solving skills are the ones now at a premium.

Like any good science, and in particular any new science, this work has generated far more research questions than it has answered.

Indeed, it is too early to say it has answered any questions. Most of us embarked on the studies with the more modest ambitions of developing new learning tools, having no expectation of finding such dramatic outcomes.

In the case of math learning, among the many factors that may be at play, all of which a (well designed) math learning game can offer, and all which are known to have a positive impact on learning, are:

Use of a human-friendly representation (not the traditional abstract symbols of math textbooks).
Focus on developing number sense and problem solving ability.
High level of engagement.
Instant feedback (both positive and negative).
Steady flow of dopamine—known to have positive impact on memory formation and consolidation.
Learning through failure—in a playful, safe environment.
“Failure” treated—and regarded—as “not yet succeeded”.
Constant sense of “I can do this on the next try.”
Lots of repetition—but at the demand of the student/player.
Student/player is in control.
Student/player has ownership.
Growth Mindset—good games encourage and develop this. (This is the important notion Carol Dweck is famous for.)
Fluid intelligence (Gf)—games require and develop this. (Loosely speaking, this is the ability to hold several pieces of information in the mind at the same time and reason fluidly with them.)

I have written about many (not all) of the factors listed above in a series of video-game learning articles in this blog. (Starts here.) Taking a broader perspective than mathematics, the many writings and video interviews on games and learning by James Paul Gee have much to say that can help us understand how those factors and others can effect learning.]

So, as of now we what we have are a scientifically sound method to conduct experiments at scale, some very suggestive early results, and a resulting long and growing list of research questions, all of which are testable. The sure looks to me like the beginnings of a genuine science of learning.

Selected sources
Berkowitz, Schaeffer, Maloney, Peterson, Gregor, Levine, Beilock, 2015. Math at home adds up to achievement in school, Science 09 Oct 2015: Vol. 350, Issue 6257, 196-198.
Kiili, Devlin, Perttula, Tuomi, 2015. Using video games to combine learning and assessment in mathematics education, International Journal of Serious Games, October 2015, 37-55.
Pope & Mangram, 2015. Wuzzit Trouble: The Influence of a Digital Math Game on Student Number Sense, International Journal of Serious Games, October 2015, 5-22.
Popovic, 2014. “Learning basic Algebra by playing 1.5h”. Center for Game Science, Uni of Washington.
Riconscente, 2013. Results From a Controlled Study of the iPad Fractions Game Motion Math, Games and Culture 8(4), 186-214.
Wendt & Rice, 2013. “Evaluation of ST Math in the Los Angeles Unified School District”, report by WestEd.

The Wuzzits – Free at Last

In which the word free has several meanings.

SEPTEMBER 3, 2013. As regular readers of this blog will know, I’ve been looking at, thinking about, reflecting on, writing about, and playing video games for many years. I’ve also been working on creating my own video games, working with a small group of highly talented individuals at a company I co-founded a couple of years ago, InnerTube Games, to create high quality casual games that embody mathematical concepts and procedures in a fundamental way.

[ADDED LATER: We renamed the company BrainQuake (brainquake.com) ]

Earlier this year, in an article in American Scientist magazine, I said a little bit about the simple (though to some surprising) metaphor for learning mathematics that guides our design, and provided a screen-shot first glimpse of our pending initial release: Wuzzit Trouble. I also discussed a few other video games that adopted a similar approach to the design of games designed to develop mathematical thinking ability – rather than the rote practice of basic skills that the vast majority of “math ed video games” focus on.

Last week, a bit later than the release date published American Scientist, we were finally able to release our game, Wuzzit Trouble. In our game, the aim is to use increasingly sophisticated analytic thinking to help the cute little Wuzzit characters break free from the traps they have got caught in.

When the game broke free from Apple’s clutches, a free download was all that was required for players from ages 8 to 80 to get to work freeing the Wuzzits. That’s three uses of “free”. A fourth was our approach broke free of the familiar tight binding between mathematical thinking and the manipulation of symbols on a page. (See my February 2012 post on this blog.)

One of our greatest worries was that many people think that mathematical thinking is the manipulation of symbols on a page according to specific rules. (My Stanford colleague, Professor Jo Boaler, has studied this phenomenon. See for example, my account of her work in an article for the Mathematical Association of America.) For anyone with that view, our game would not appear to offer anything particularly new or different. That would mean they would fail to grasp the power of our design metaphor, as I had described in my American Scientist article and in a short video (3 min) we released at the same time as the game.

That will likely be a problem we continue to face. It will, I fear, mean that some people we would like to reach will dismiss our game. (On one remarkable occasion, an anonymous reviewer of a funding application we submitted to cover the development costs of our game, after playing an early prototype, declared that there was not enough mathematical content. All I can say to anyone who thinks that is, give the game a try and see how far you get – see later for the fine print that accompanies that challenge.)

Fortunately, the first review of our game, published in Forbes on the day Apple released it in the App Store, was written by an educational technology writer who understood fully what we are doing. That initial review set the tone for many follow-up articles. We were off to a good start.

We were also greatly helped by Apple’s decision to feature our game, which appeared front and center on the App Store website for educational apps.

Apple features Wuzzit Trouble on its release

Other websites that track and report on the apps world followed suit, and before long we found ourselves in the Top Fifty of new educational apps. People seemed to “get it.”

Presenting a mathematics video game that does not have equations, formulas, or other symbolic mathematics all over the screen is just one way we are different from the vast majority of math learning games. Another is that we built the game to allow players of different ages and mathematical abilities to be able to enjoy the game.

As I describe at the end of another short video, if all you want to do is free all the Wuzzits, all you need is basic whole number arithmetic, which means the game can provide a young child lots of practice with basic number work.

But if someone older wants to get lots of stars and bonus points as well, much more effort is required. (Just check out the solution to one of the puzzles I describe on that video.) This is what we mean when we say Wuzzit Trouble provides a challenge to any player between the ages 8 and 80.

But this is already way too much text. Writing about video games is like writing movie reviews. Both are designed to be experienced, not read about. Just download the game and try it for yourself. And if you are so inclined, take me up on The Math Guy Challenge.

For more details about InnerTube Games and Wuzzit Trouble, visit our website: http://innertubegames.net.

How to design video games that support good math learning: Level 6

Show me the action!
Part 6 of a series

APRIL 6. 2012. Whether you view mathematics as a collection of procedures or a way of thinking (see my last post), math is something you do. Or cannot do, as the case may be.

When I meet people for the first time and tell them my profession, they frequently reply, “I never could do math.” What they never say is, “I don’t know math.” Everyone, whether mathematically able or not, realizes that math is not stuff that you know, but an activity you do.

Of course, sleeping, sitting on the beach daydreaming, and watching TV are also activities, but they are passive activities. I am using the word “activity” in its stronger sense. That stronger sense certainly includes mental activity. As a simple rule of thumb, you know something is an activity in my sense if doing it makes you tired. By that metric, math is one of the most strenuous activities I know — and I’m one of those people who spend their weekends cycling over mountain passes for seven or more hours at a stretch.

What is the most efficient way to learn how to do something? We all know the answer, and we did so long before Nike turned it into a commercial slogan: Just do it!

If you want to learn to ride a bike, drive a car, ski, play tennis, play golf, play chess, play the piano, and so forth, you don’t start out by attending a lecture or reading a book. Those can be useful supplements when you have reached a sufficient level of proficiency and want to get better. But learning from a lecture or a book require interpretation and assimilation of incoming information (a static commodity), and that in turn requires sufficient prior understanding. No, what you do is start to do it.

Very likely you don’t start out doing it unaided. You seek guidance, from a parent, relative, friend, instructor, professional coach, or whatever. And in the course of helping you learn, that person may well give you instructions and advice. But they do so in the course of you performing the activity you are trying to learn, when what they say makes sense and has immediate, recognizable value.

With everyone, it seems, in agreement that mathematics is an activity, and given our collective experience that mastering an activity is best achieved through doing it, we have to ask ourselves how mathematics education has come to be dominated by the math textbook?

Though there is an argument to be made about the self-interest of textbook publishers, the fact is that mathematics instruction has been delivered through textbooks since the subject began. Archimedes’ Method, Euclid’s Elements, al Khwarizmi’s Al-Jabr, Leonardo of Pisa’s Liber abbaci, and on throughout mathematical history, the symbol-heavy, written text has been the primary vehicle for storing and disseminating mathematical knowledge.

Why? Because putting words and symbols on a flat surface was the only technology available for the task!

But video games — or rather, video game technologies — provide us with an alternative. The digital framework in which a typical video game is embedded is dynamic and interactive, and can provide the experience of moving around in a 3D world. In other words, video game technologies provide platforms or environments suited (by design) for action. Which makes them ideal for representing and doing mathematics (an activity).

The task facing the designer of a video game to provide good mathematics learning experiences is to represent the mathematics using the natural affordances of the medium. This means putting aside the familiar symbolic representation. My own experience, having been doing this for over five years now, and working with others doing the same thing, is that it is initially very difficult. People have been using symbolic representations or one form or another for several millennia and that has conditioned how we think of mathematics. But it is worth making the effort, because the potential payoff is massive: it will circumvent the Symbol Barrier, which I discussed in the third post in this series.

In addition, by representing the mathematics in a medium-native fashion, we will minimize, and in some cases eliminate, the degree to which “doing the math” detracts from the game mechanics. For some students — the ones with a natural affinity to mathematical thinking — this is not a big deal, since they will gain satisfaction from solving the mathematical problem, but for many students, advancement in the game will be the main driver.

I should stress that what I am advocating is not watering down mathematical thinking to a “video game version” of mathematical thinking. At a conceptual level, it is the same thinking; only the representation is changing. Once the student has mastered mathematical thinking presented in “video-game language”, a teacher could use that experience as a foundation on which to base instruction in the symbolic representation of the same concepts and thinking.

That last step is an important one, in part because mastery of symbolic mathematics is what is required to perform well in standardized math tests, and regardless of your views on the educational value of such instruments, they are currently a fact of life for our students. But there are two other reasons why it is important to transition the students to symbolic mathematics. First, mastering multiple representations greatly assists good conceptual learning, and the abstract symbolic representation, by virtue of its abstractness, is particularly powerful in that regard. Second, the symbolic representations make it much easier to apply mathematical thinking to a wide variety of new problems in novel domains.

I’ll pursue these ideas further in subsequent postings. In the meantime, let me leave you with three examples of video games that present mathematics in a medium-native fashion: Motion Math, Number Bonds, and Jiji. Notice that in each case the mathematical concept involved is represented in a medium-native, and dynamic fashion. The player interacts directly with the concept, not indirectly via a symbolic representation, in the same way that a person playing a piano interacts directly with the music, not indirectly via a symbolic musical score.

To my mind, this is one of the most significant, and potentially disruptive benefits of using video games in mathematics education: they offer the possibility of direct manipulation of mathematical concepts, thereby circumventing the symbol barrier. Achieving this direct connection to the concepts is not easy. Those three games may look simple. Indeed, to the player, they are simple, and that is the point! But I know for a fact that all three took some very smart folks a lot of time and effort to produce. That’s usually the case with any tool that looks simple and works naturally. Designing simplicity is hard.