During the 1920s, West Electric were interested in how they could improve productivity. So they sent a team of researchers to their Hawthorne Works factory just outside of Chicago. Over a period of 8 years, the research teams experimented with different lighting levels and measured worker output.
As expected, when changes were made to lighting intensity during the experiments, productivity increased. But the increased performance could never be repeated once the experiments had finished. The boost was not, it seemed, down to the light changes, but simply a positive response from the workers to the fact that they were being observed.
Twenty years later, sociologist Henry A Landsberger coined the term ‘the Hawthorne Effect‘ to describe this phenomenon, and researchers in the social sciences have been wrestling with it ever since.
When applied to classroom observation, I believe that the Hawthorne Effect can be devastating. This brings into question, then, any educational research which includes this methodology in its research design. That’s what I want to talk about here.
The literature collected under the umbrella of Educational Research varies massively in the questions that it seeks to answer and the methods employed to answer them. This is not simply a qualitative/quantitative paradigm war; flick through any academic education journal and you will find theoretical critiques, ethnographies, policy analyses, psuedo-psychological studies, epistemological proposals…the list goes on and on.
Often, however, the researchers involved will be interested in observing classrooms. The trouble here, of course, is that within any classroom you find two groups of people who have become masters of acting in a completely different manner when they know that they’re being observed.
For instance, I can lip-read a group of students sharing the antics of their Sims avatars when they are supposed to be identifying rhetorical questions and discussing their effect on the reader. By the time I make it over to their table, however, they have miraculously switched mid-sentence to ‘the emotional impact of the unanswered question’.
And the second guilty party? Teachers. It’s hardly a secret that after the dreaded ‘call’, teachers stay until midnight trying to make sure that their classrooms, books, lessons and teaching are as far from what they normally look like as possible. In other words, the observations are invalid.
The fact that observations are invalid as a means of collecting information about teaching and learning was the first nail in the coffin of Ofsted graded observations.
The second nail came after the publication of Policy Exchange’s ‘Watching the Watchmen’ report, which unearthed a little read but remarkable piece of research from Michael Strong, John Gargani & Ozgeshowing Hacifazlioglu (2011), which showed when it came to judging lessons, people were terrible at making an accurate judgment. That is to say, observers’ judgements didn’t tally with the actual Value Added to the pupils during the observed lesson. Their chances of getting a grade ‘correct’ was about 50% – or no better than tossing a coin.
So, as well as suffering from being invalid, classroom observation is unreliable.
All of this sounds fairly damning, and after articles and blogs much better than this one, graded observations were emphatically dumped by Ofsted.
But if observation is such a lousy method of gathering data, then what of all of the educational research that relies on this tool? Many of the same arguments stand up. Teachers and students act in ways that they wouldn’t normally; they are practiced experts when it comes to behaving in a manner that they hope will impress the visitor with the clipboard.
So what we are left with is researchers making claims about what might work in your average classroom anywhere in the country, based on findings which we know are heavily distorted by the Hawthorne Effect (sometimes referred to, appropriately, as the ‘observer effect’).
This goes some way to explaining the fact that ‘everything works’ in education research. Interventions always have a positive effect, but how much of this is down to the fact that the students and the pupils know that they are being observed, and so raise their game?
The upshot may be that the most effective intervention we can make in education is simply to observe classrooms, regardless of what it is that the observers are supposedly looking for.
This presents a difficult conundrum for educational researchers. We don’t simply want to rely on cold and detached numbers when investigating what works in education. The complex human interaction that happens in every classroom is what we want to probe and dissect, to try and find common themes and unusually effective practices.
Beware, though, the fact that education suffers the same Heisenberg Uncertainty Principle that plagues quantum physicists.
By observing something, you change it.