Andrew Maul presenting as part of "Fast & Curious" in February 2017
Andrew Maul, Assistant Professor in the Department of Education, likes to share the story of Phlogiston, the combustible substance found in every flammable object…at least according to scientific belief from roughly 1670-1790. Scientists spent a lot of their research and thinking trying to prove what’s now called oxidation was the release of phlogiston. “It was a theory that was in many respects successful, in spite of being false,” Maul delightedly points out. “In the end, progress depended not only on advances in scientific methodology, but on a certain humility, a willingness to let go of old ideas. I think we social scientists may be in a similar position to Phlogiston theorists of the eighteenth century.”
Maul, who calls himself “a meta-social scientist—my scholarship is about the ways we do scholarship,” currently is the PI on a Spencer Foundation grant entitled “Reexamining the Philosophical Foundations of Educational Measurement.” It’s his second grant from Spencer, as he also had a postdoc fellowship that explored similar themes (“Improving the Measurement of Affective Attributes”). To oversimplify his goals, he claims his aim is to discover “how we [education researchers] can be more rigorous and more coherent.
“The social sciences have attempted to claim scientific rigor because the public tends to trust people with A) Ph.D.s and B) numbers. Especially numbers,” he says. “Surveys are ubiquitous and people make huge claims. There are real stakes being attached to these things, but what really justifies the claims we want to make about people?”
Maul suggests we take a step back, for he believes “statistics have their place, but without a theoretical grounding they can lead us astray.” He points to the early 20th century origins of the social sciences, and its roots in logical positivism and behaviorism, as something 21st century researchers need to reassess. “The legacy of those ways of thinking, while largely rejected in the formal literature, still remains,” he says. “Some social scientists seem to think that if you have a number-generating procedure you have a measure by fiat – that’s violently out of step with the way the scholarship on measurement in philosophy, metrology, and nearly every other scientific discipline.”
To help people re-think social science research, Maul, along with Luca Mari, a metrologist from the University Carlo Cattaneo in Italy and Mark Wilson, Maul’s former graduate advisor at UC Berkeley, are working on a book. “The goal is to open up conversations,” he explains. “It will be a comprehensive account of what measurement is and how it works, in an interdisciplinary way.”
Here’s how the summary for his Spencer Grant puts it: “The word measurement is now nearly as ubiquitous throughout the literature on educational research and policy, and in the social sciences more generally, as it historically has been in the physical sciences. It is regularly claimed or implied that scores on academic tests or surveys constitute measures of student knowledge, skills, and abilities, attitudes, motivations, perceptions, and personality factors. Though it is commonly recognized that the meaning of measurement must be different in in the social and physical sciences, it is less commonly recognized that social scientists have collectively failed to provide a coherent alternative account of the meaning of measurement, leaving open the possibility that knowledge claims made on the basis of putative measures of human attributes are either false or pseudoscientific. Dr. Maul and colleagues are attempting to cultivate greater dialogue on the topic of measurement across different scientific and philosophic disciplines, with the hope that both social and physical measurement theorists may benefit from contact with other fields. In psychometrics, in particular, a greater focus on conceptual clarity could serve to supplement the almost exclusive focus on mathematical modeling and software development currently found in the flagship journals.”
While Maul half-jokes, “I could have picked a safer career path…,” he follows that line with the statement, “We have to be much more clear about properties of people like grit and self-control. The sooner we replace old ideas with better ones, the faster we will make progress.” Just like scientists at the end of the 18th century doing the work to make sure Phlogiston went up in smoke.