Matrix Evidence

A wise sage once said, ‘the plural of anecdote is not evidence’. Many who work producing evidence for policy will have faced the frustration of having decision makers weighting stories more highly than good quality empirical evidence.

However, a blog I came across this week makes a very reasonable point about how empirical scientists can sometimes use anecdotes very effectively to illustrate points supported by robust data. I find myself agreeing with the author – effective evidence for policy need not always avoid anecdotes. We should recognize their limitations of course, but at the same time recognize their power to reinforce, for certain audiences, messages that have been drawn from robust empirical evidence. It’s by no means an easy feat to get the right mix, and in some instances, for some audiences, the use of anecdotes is inappropriate. But as any good communicator will tell you, stories can be a very useful tool in getting messages across. As scientists working in the evidence for policy arena, we ignore that fact at our peril.

A Centre for British Teachers (CfBT) report, ‘Instinct or Reason: How Education Policy is Made and How we Might Make it Better’ concludes that since the 1970s, education policy has been influenced by political ideology, prime ministerial preferences and political advisors rather than sound research evidence.

No surprises there then. The report goes on to suggest that evidence has failed to influence policy because of mistrust in academic researchers. Actually, it probably has more to do with the poor quality of too much educational research. That was the conclusion of the 1998 Hillage report. At the time, educational research was described as ‘dross, second-rate and gobbledegook’, not by politicians, but by education research professionals. A Times Higher Education article at the time of Hillage included this paragraph:

The research pouring out of university departments of education is blighted by a proliferation of “dross that should never have seen the light of day”, says Oxford professor Richard Pring. Half of what is published is “second rate”, says Newcastle professor James Tooley, after a study of the top research journals in the field. Much is “gobbledegook”, says Alan Smithers, of Liverpool University.

Judging by a response to the CfBT report from Peter Mortimore in a Guardian article, things have probably not changed that much. The former director of London University’s Institute of Education warns against having a body that regulates the excellence of educational research on the grounds that it ‘might seek to impose an orthodox view of research’. He goes on, ‘as countless scientific discoveries have show, researchers must not be corralled if they are to produce revolutionary thinking’. Forgive me for my cynicism, but does that not sound dangerously like an excuse for more dross?

Let’s test the hypothesis. Anyone out there care to start a count of the countless? I’m looking for examples of landmark scientific discoveries that resulted from procedures that were inconsistent with orthodox scientific methods.