« A Certain Blindness | Main | Urban Art and Education (Part 1) »

Epidemiological Methodology and Education: The Fun Stuff

[Note: I have recieved some great critiques regarding this post, including the thought that I may be conflating some of my concepts. See my response at the end of the post.]

Preface

This TED Talk on education data that strives to test students' ability to adapt to change and solve real, unfamiliar problems. Andreas Schleicher and his team have done incredible work. I am going to criticize their great work in this post, and I want to be clear that I see serious critique as the highest respect we can pay to work of any kind.

It means that we take the work so seriously that we are willing to devote our own energies to try to improve it. I'm not writing this preface as an apology, but mostly to make a point about critique that is often not recognized.

So, on with it...

A Bit of Background

I highly recommend that you watch the talk above--this post will still make sense, but won't hit home in the same way if you don't.

For those of you who can't listen to the audio/video of the talk:

Andreas Schleicher heads the Program for International Student Assessment (PISA) at the Organization for Economic Co-operation and Development (OECD). What it means is: He's designed a test, given to hundreds of thousands of 15-year-olds around the world (the most recent covered almost 70 nations), that offers unprecedented insight into how well national education systems are preparing their students for adult life. As The Atlantic puts it, the PISA test "measured not students’ retention of facts, but their readiness for 'knowledge worker' jobs—their ability to think critically and solve real-world problems."

The results of the PISA test, given every three years, are fed back to governments and schools so they can work on improving their ranking. And the data has inspired Schleicher to become a vocal advocate for the policy changes that, his research suggests, make for great schools.

It's very interesting and useful research; however, it's heavily overstated.

Their approach has been to take epidemiological methodology and apply it to a social science (education research). Unfortunately, by definition, this methodology cannot yield valid conclusions. They're not alone on this though; they're part of the vast majority. This problem has been plaguing the physical and social sciences in a very aggressive manner over the past 40+ years.

 

Epidemiological Methodology

Epedemiology began as a methodology for studying and predicting the transmission of communicable diseases (primarily bacterial and viral). Because these diseases are passed through physical means, and because we understand the organic mechanism of action through which the bacterium or virus causes disease, we can draw valid conclusions/make predictions based on epidemiology.

To clarify: if we know that Cholera bacteria cause particular symptoms in all healthy humans, we can say that the physical relocation of Cholera from [Non-immune Person A] to [Non-immune Person B] will cause the same symptoms. We can say this because humans are nearly genetically identical, though they may vary widely in terms of phenotypic expression.

When using epidemiological methodology, we need causality to make any sort of valid conclusion. No exceptions, and "close" doesn't count. If we don't have causality, the furthest we can state a claim is in the form of a testable hypothesis.

Communicable disease are just about as close as we can get to showing genuine causality, as opposed to correlation, so we don't need any further investegatory steps to make valid conclusions/recommendations. Side note: for most of modern history, on of the only other things that we could really claim proper causality for was Newtonian physics...but Bohr, Einstein, Planck, etc. changed that.

We are able to draw valid conclusions from epidemiology, as the study of transmission of physical disease factors, because of the strength of the correlation coefficient, which is effectively 1 in the case of communicable disease.

What Does This Have To Do With Education Research?

Since other sciences have adopted epidemiological methodology, they have failed to recognize the above fact about epidemiological methodology.

Because correlation coefficients in other sciences are not 1, we cannot draw conclusion based on this sort of comparative/epidemiological methodology. Period. We can't even say, "Well, it's strong enough to make a compelling case for conclusion X." There are too many confounding factors. All we can say is that the correlation warrants the formation of a hypothesis, based on a proposed mechanism of action, that will then need to be tested.

Schleicher does a good job guarding against outright, cross-contextual conclusions, but the problem is that he's still looking at epidemiological factors and not making it to testing via proposed mechanisms of action. (Note: a proposed mechanism would NOT just be a context-appropriate blend of epidemiological factors.)

In nutrition, epidemiology is misused to draw conclusions (aka, recommendations) about systems of eating based on cross-context/country studies. Unfortunately, these conclusions/recommendations are, by definition, invalid. Following an epidemiological study, the next step requires that we propose and test a mechanism of action in order to reach (or work towards) a conclusion. In the case of nutrition, this takes the form of biochemical lab tests based on an understanding of organic/biochemical mechanisms.

To clarify things, let's look at an example of each kind of study. Pay close attention to the kind of thing that each study is trying to accomplish:

Example of epidemiological study: "Risk factors for subclinical atherosclerosis in diabetic and obese children."--this is the sort of study that you cannot use to make conclusion/recommendation.
Goal of study: "We aimed to evaluate early signs of atherosclerosis and investigate for predisposing factors in children and adolescents affected by type 1 diabetes."
Example of study looking at a mechanism of action: "Mechanisms of β-Cell Death in Type 2 Diabetes"--this is the sort of study that tests a mechanism of action. A series of studies like this would allow us to reach a valid conclusion/recommendation.
Goal of study: "We will show that pathways regulating β-cell turnover are also implicated in β-cell insulin secretory function. Depending on the prevailing concentration and the intracellular pathways activated, some factors may be deleterious to β-cell mass while enhancing insulin secretion, protective to the β-cell while inhibiting function, or even protective to the β-cell while enhancing function."

Notice that the first (epidemiological) study, if combined with other studies, at best allows us to pose a hypothesis that would become the subject of the tests in the second (mechanism) study. Only after we have verified a mechanism of action can we then reach a conclusion/recommendation (which would then be subject to review, verification, etc.).

Note: The above two studies are not the best examples because they concern biologically distinct disease states, so we wouldn't stack them in the progression I present above, but it's close enough to make the point.

But What About Education?

An example of an education study using epidemiological methodology? The video above.

An example of an education study that looks for mechanisms of action? I've never seen one. I've tried to lay the groundwork for this in my thesis because I have not seen it anywhere in education research, nor much of any social science research in general.

It's easy enough to come up with examples of mechanisms in the physical sciences if you're science literate: tidal forces, β-cell death, heat transfer, etc. But what would a mechanism of action look like in the social sciences?

An example of work that comes close to reaching for a mechanism of action in the social sciences is, perhaps oddly, Peircean Semiotics. Whether Peircean Semiotics is helpful or not, it is still a proposed mechanism of action. It represents a relatively easy-to-grasp example of the kind of thing I mean when I point towards mechanisms of action in the social sciences. It is a mechanism in part because it is not a nod towards particular populations or their features, but rather a look at certain rules that these populations are subject to.

###

Amendment:

Elsewhere, I received the critique that I'm conflating statistical methods with epidemiology, and that what I'm talking about actually has nothing to do with epidemiology. Here is my response:

I should be more clear: problematizing this through the lens of epidemiology is helpful because it points out the shortcomings in such a way that we have a very wide range of possible solutions immediatley available to us. Statistics alone is too broad of a scope to look at the problem, because it is a tool that is used across all sorts of different applications. In other words, yes, both use statistical methods, but if we're interested in ameliorating a specific issue, we need a context-specific methodology that allows us to do so. Statistics-broadly does not give us any such context-specific method, but epidemiology provides an analogical, context-specific method that we can learn from.

Any practical recommendation we make based on data is not part of statistics-proper. We may make statistical inferences, and even this is falls at the furthest reaches of what we might still call statistics, we have still not yet made practical recommendations for actually people to follow. Once we start (implicitly or explicitly) making practical recommendations, we're not only using statistical data, but we now require much context-specific methods that are out of the realm of statistics. This is why it is helpful to look at this problem through the lens of epidemiology--epidemiology is a practice that often misuses statistics in similar ways to education research, but the best biochemists know how to make statistical data valuable in a methodologically valid manner.

[Click here and I'll let you know when I publish future posts.] 

Reader Comments (5)

Great post, Nathan. I have a question that maybe you could cover in a part 2. Do you think it's ever possible to prove causation in education research? Even if you looked at something as simple as "does raising teacher x's salary improve classroom y's scores on test z?", I think you will still run into a whole host of problems (I'm thinking of confounding variables and low statistical significance). In other words, I can't imagine two variables found in education that could produce a strong correlation coefficient.

Even if this is not possible, you're point is still well taken. There's a world of difference between suggesting a hypothesis for further testing and claiming causation.

Also, something that you never touched on was the validity of the PISA data. I think the temptation is to look a country's PISA data and make a claim about it's education system as a whole. But where's the proof that PISA's data actually represents the types of effects we're looking for in the first place? The best you can do is "these factors may have influenced these PISA scores in this way." Thoughts?

February 23, 2013 | Unregistered CommenterPaul Bernard

You are right that empirical data (correlational analysis etc) in itself is never enough to make hard causal claims in educational research. The causal mechanism (if any) behind the correlation is often a black box. In order to understand causal mechanisms, you need theory. And yes indeed: in many cases there will be different competing theories trying to explain the same empirical finding. This should not discourage us. I still believe that empirical analysis, such as the analysis of the PISA data helps to improve the public and scientific debate on education.
We will have to accept the uncertainty of educational research. The comparison between epidemiology and educational research does not hold, because epidemiology is not interfered (so much) by culture. e.g. you should not try to uncover the 'universal' effect of class size on educational performance, because it does not exist. It is to a large extent mediated by culture and context.

February 23, 2013 | Unregistered CommenterAnton Derks

Anton,
Most of this, I agree with. One of the problems I'm trying to point out though is that the causal mechanisms are a currently a black box precisely because we have not developed a systematic way to work with social concepts in the same way that we systematically take up hard-scientific endeavors--not because causal mechanisms in the social sciences are inherently black-boxy. The fact that "competing theories" are the norm in social sciences points this problem out; a problem that does not plague the hard sciences in the same way. Further, the tradition that has developed to "chalk it all up to culture and subjectivity" seems to be the contemporary equivalent of the God solution that even great minds like Newton eventually reached: if we don't understand something, it must be God's design (or in contemporary language, it must be because of the complexity of culture or the variance of subjectivity).

The Vienna Circle and the logical positivists took a stab at developing a systematic way to deal with concepts but failed, I think because they were so reliant on metaphysics (ironically) and not enough on the world.

I do not agree that we have to accept the uncertainty of education research. The further development of scientific method (meta-science, if you will) can only lead to more lucid and intentional processes, hypotheses, conclusions, etc. Scientists don't need to be focused on this (otherwise, nothing would get done), but there are still SO many frontiers in scientific methodology. As those methodologies progress, we will see less reliance on "explaining things by culture/subjectivity" and more explanations that look like the hard sciences.

Finally, empirical analysis is of the utmost importance, and the analysis of the PISA data is, as I said, very good and helpful research. It cannot, however, tell us anything about what our education practices should be in the same way that diet-related disease epidemiology cannot tell us what our eating practices should be (though many seem to be under the false impression that it can).

February 23, 2013 | Registered CommenterNathan Pai Schmitt

Paul,

From "The Structure of Scientific Revolutions," by Thomas Kuhn:

"...It makes little sense to suggest that verification is establishing the agreement of fact with theory. All historically significant theories have agreed with the facts, but only more or less. There is no more precise answer to the question whether or how well an individual theory fits the facts. But questions much like that can be asked when theories are taken collectively or even in pairs. It makes a great deal of sense to ask which of two actual and competing theories fits the facts better."

Science, whether physical or social/conceptual, does not strive to verify facts as the common (mis)conception of it goes. Rather, its job is to produce explanations that seems to fit our experiences/conditions/endeavors/etc. better than previous attempts to explain. It is not an exploration of objectivity, nor is it an exercise in subjectivity. Science and its methodologies are tools that we use for human purposes. Talk of subjectivity/objectivity in science are relevant on more micro-levels in scientific practice because it is helpful in framing particular sorts of studies...but the usefulness of these concepts is pragmatic only, not ontological/metaphysical.

So my point here is that causality is not the goal. In discussing causality, I'm pointing out that IF (and it's a huge "if") we're drawing conclusions from epidemiology-like methods, we cannot make claims without causality.

After thinking about this more, I'd actually like to make a small amendment to this post. I said above that, "Communicable disease are just about as close as we can get to showing genuine causality, as opposed to correlation, so we don't need any further investegatory steps to make valid conclusions/recommendations." I now think this is wrong. The reason we can make claims about communicable diseases in this way, I think, is because it simply works better than any other method we have to do so. When newer, better methods come along, of course we replace the old ones. A perfect example of this would be Newton to Einstein to quantum theorists.

This process is something that is widely recognized in the physical sciences, but people tend to be resistant to it in the social/conceptual sciences. As I implied in my previous comment reply, using culture/subjectivity/vast complexity to "explain" the thus-far-unexplained is not an approach congruent with progress in the sciences. We need a way to legitimately explain things, and we need not to dismiss things we cannot yet explain with God, culture, and so on.

February 23, 2013 | Registered CommenterNathan Pai Schmitt

Nathan,
Thank you for your reply. However, I do not agree. Culture is not The Unexplainable. Culture is a context that mediates the relationship between A and B (for instance between class size and educational outcomes). The challenge is trying to understand the way in which culture mediates social processes. It is a complex endeavor. I admit that progress in this field is very slow, but there are some interesting developments, for instance in cultural sociology (Jefffrey Alexander, strong program, ..).
Culture is something real, it is not a 'bias'. It has to do with signs, symbols and interpretation (hermeneutics). Water will boil at 100°C, regardless of culture. In educational research, most universalistic claims ("the effect of A on B") are false, but contextualized claims can be true.
There are also interesting development in evaluation studies (http://www.amazon.com/Mind-Gap-Perspectives-Evaluation-Comparative/dp/1412810752).

March 16, 2013 | Unregistered CommenterAnton Derks

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>