5 Big Ideas That Don't Work In Education

There are few household names in education research. Maybe that in itself constitutes a problem. But if there was an Education Researcher Hall Of Fame, one member would be a silver-haired, plainspoken Kiwi named John Hattie.

Hattie directs the Melbourne Education Research Institute at the University of Melbourne, Australia. He also directs something called the Science of Learning Research Centre, which works with over 7,000 schools worldwide.

Over the past 28 years he has published a dozen books, mostly on a theory he calls Visible Learning. His life's work boils down to one proposition: To improve schools, draw on the best evidence available.

Obvious? Maybe, but it's rarely honored in reality, Hattie claims. "Senior politicians and government officials clearly want to make a difference," he says. "But they want to do this, that and the other silly thing which has failed everywhere else, and I want to know why." In a new paper, "What Doesn't Work In Education: The Politics Of Distraction," published by Pearson Education, Hattie takes on some of the most popular approaches to reform.

Small classes. High standards. More money. These popular and oft-prescribed remedies from both the right and the left, he argues, haven't been shown to work as well as alternatives.

Hattie doesn't run his own studies. Nor does he analyze groups of studies on a single variable, a technique called meta-analysis. He goes one step further and synthesizes the findings of many meta-analyses, a kind of meta-meta-analysis.

Over the years, he has scrutinized — and ranked — 1,200 different meta-analyses looking at all types of interventions, ranging from increased parental involvement to ADHD medications to longer school days to performance pay for teachers, as well as other factors affecting education, like socioeconomic status. He has examined studies covering a combined 250 million students around the world.

The good news, he says, is that most education reforms tested in published studies show at least some positive effect (this should not be surprising, because studies that show no effect or negative effects are less likely to be published).

If you are the kind of person who finds certain graphs sexy, beholding Hattie's ranking of educational effect sizes will be exhilarating.

The average effect, across all the studies he's analyzed, is 0.4. standard deviations. This average also happens to translate — roughly — to the amount of progress a student can be expected to make in one year of school. Hattie believes that all educational reforms should concentrate on interventions with proven effects that fall above that line.

In his ranking, socioeconomic status has an effect size of 0.57, meaning that a student growing up in poverty may be expected to perform roughly a year and a half behind an otherwise similar student growing up more wealthy.

Putting televisions in the classroom, on the other hand, has an average negative impact of -0.18. Holding students back a grade really does hold students back, with an effect of -0.16.

"The problem is there are a lot of effects that are very small," he says, while others are huge. And yet, he says, "We never have a debate of relativity — why are we spending billions on things that have small effects?"

Technical Challenges

Hattie's grand unified theory is simple — maybe too simple. Critics have taken issue with his approach to research, the precision of some of his calculations, even his grasp of concepts as basic as probability.

"Meta-analyis is relatively new in education, and ... particularly problematic," says Dylan Wiliam, professor emeritus of educational assessment at the Institute of Education, University of London, and an expert on assessment.

He argues, for example, that averaging together studies done on students of different ages, in different settings, with different kinds of interventions and different measures of outcomes may produce entirely misleading results.

There's a danger, Wiliam says, of mushing good studies together with bad ones, or comparing apples and oranges.

"In education, meta-analysis presents a number of significant technical difficulties," he explains. "Some of these are unavoidable but Hattie does not mention these."

Others, Wiliam adds, "are avoidable, but Hattie does not avoid them."

"The synthesis approach is not an established method," agrees John O'Neill, director of the Institute of Education at Massey University in New Zealand. O'Neill is a coauthor of a 2009 paper critical of Hattie's work, titled "Invisible Learnings?"

At the same time, he acknowledges, Hattie's work "has had a profound effect on education policy and practice globally."

Many of Hattie's basic observations have been upheld by other researchers. And he and his organization continue to advise and influence governments and school leaders all over the world.

Here are five of the most common policy ideas that, he argues in his new paper, are wrongheaded — and the alternatives Hattie suggests.

1. Achievement standards. "It seems very sensible. You set up minimum standards you want students to reach; you judge schools by how many reach them. But it has a very nasty effect," Hattie tells me. "All those schools who take kids in difficult circumstances are seen as failures, while those who take privileged students and do nothing are seen as successful."

By the same token, it seems to make sense to set achievement standards by grade level, but the further along students get in school, Hattie points out, the more of them are performing either behind or ahead of the schedule that's been set.

The alternative: a focus on growth and progress for each student, no matter where he or she starts.

2. Achievement tests. High-performing schools, and countries, don't necessarily give more standardized tests than low performers. They often give fewer.

The alternative: testing that emphasizes giving teachers immediate, actionable feedback to improve teaching.

3. School choice. Many education reformers tout school choice as a tool for parent empowerment and school improvement through competitive pressure. But Hattie says his research shows that once you account for the economic background of students, private schools offer no significant advantages on average. As for charter schools? "The effect of charter schools, for example, across three meta-analyses based on 246 studies is a minuscule .07," he writes.

The alternative: teacher choice. In the United States, variation within schools accounts for 70 percent of the differences in scores on the international PISA exam, while variation between schools makes up the rest. Hattie argues that if parents had the right to select the best teacher in a given school, that could truly be empowering. It would also be challenging to implement.

4. Class size. This has been one of Hattie's more controversial claims. In the U.S., groups such as Class Size Matters are dedicated to the proposition that fewer students per teacher is a recipe for success. This, Hattie argues, would come as a surprise to Japan and Korea, two of the highest-performing education systems in the world, with average class sizes of 33. Russia is the outlier in the other direction, a below-average performer with average classes around 18.

The alternative: Hattie says reducing class size can have a positive impact. That's if teachers are coached and supported to take advantage of it by actually changing the way they teach — to collaborate, offer personalized feedback and continuously measure their impact for improvement, for example.

5. More money. $40,000 per child, from age 6 through high school graduation. That's the rough threshold for reasonable school performance, according to Hattie: Countries that spend less than $40,000, which are all poor, tend to have much lower reading scores on the international PISA exam, and their performance correlates strongly with the money they spend. But for countries above that threshold, there is almost no relationship between money spent and results earned. For example, Korea and Finland far outscore the U.S. on PISA, while spending $60,000 and $75,000 compared with $105,000.

The alternative: Money's a necessity, but more money is not a panacea, says Hattie. "We spend millions on things that don't matter, and then we get jaundiced."

Hattie's forthcoming book, in September, will present case studies of 15 schools that are implementing some of the ideas that have the strongest evidence behind them. He says many of these boil down to empowering teachers to work collaboratively and continuously improve.

"Around the world there is so much excellence," he says. "Have we got the spine to identify and grow that?"