Evidence Corner

Welcome to the Evidence blog. I'll use this space to bring you a selection of material on evidence and policy I come across. I very much hope you will find it useful, and maybe even bring your own contributions.

CfBT report on evidence-based education policy: have we learnt nothing?

Friday, June 4th, 2010

A Centre for British Teachers (CfBT) report, ‘Instinct or Reason: How Education Policy is Made and How we Might Make it Better’ concludes that since the 1970s, education policy has been influenced by political ideology, prime ministerial preferences and political advisors rather than sound research evidence.

No surprises there then. The report goes on to suggest that evidence has failed to influence policy because of mistrust in academic researchers. Actually, it probably has more to do with the poor quality of too much educational research. That was the conclusion of the 1998 Hillage report. At the time, educational research was described as ‘dross, second-rate and gobbledegook’, not by politicians, but by education research professionals. A Times Higher Education article at the time of Hillage included this paragraph:

The research pouring out of university departments of education is blighted by a proliferation of “dross that should never have seen the light of day”, says Oxford professor Richard Pring. Half of what is published is “second rate”, says Newcastle professor James Tooley, after a study of the top research journals in the field. Much is “gobbledegook”, says Alan Smithers, of Liverpool University.

Judging by a response to the CfBT report from Peter Mortimore in a Guardian article, things have probably not changed that much. The former director of London University’s Institute of Education warns against having a body that regulates the excellence of educational research on the grounds that it ‘might seek to impose an orthodox view of research’. He goes on, ‘as countless scientific discoveries have show, researchers must not be corralled if they are to produce revolutionary thinking’. Forgive me for my cynicism, but does that not sound dangerously like an excuse for more dross?

Let’s test the hypothesis. Anyone out there care to start a count of the countless? I’m looking for examples of landmark scientific discoveries that resulted from procedures that were inconsistent with orthodox scientific methods.

Evidence-based policy in Australia

Friday, April 16th, 2010

The Australian Productivity Commission organised a roundtable discussion on the issue of strengthening evidence-based policy in 2009. Participants included government officials, academics, consultants and representatives of non-government organisations. Keynote addresses were presented by Dr Ron Haskins, Senior Fellow of the Brookings Institution, and Professor Jeffrey Smith of the University of Michigan.

The roundtable discussed the principles of the evidence-based policy movement and reviewed how well Australian use of evidence conformed to best practice. It then considered how to improve the availability of quality evidence, and reviewed possible institutional developments to embed good use of evidence more firmly into policy-making.

The roundtable proceedings have now been published. They include papers by the speakers and a summary of the key points covered in the discussion sessions, and a series of background papers prepared by Commission staff and provided to roundtable participants.

I was particularly drawn by a paper entitled ‘Evidence-based policy: summon the randomistas’ by Andrew Leigh of the Research School of Economics at the Australian National University

Obliquity – more debate on evidence-based policy

Friday, March 26th, 2010

In a book to be published this month, “Obliquity – how our goals are best pursued indirectly” British economist John Kay looks at the reasoning behind the notion of evidence-based policy and finds it wanting.

Kay defines evidence-based policy making as a process of determining the goals of a policy intervention, and then looking for evidence to determine the most effective means of delivering those goals. This, he claims, is not a good description of how policy decisions are made in reality, or could ever be.

Having set up his straw man, Kay goes on to knock it down, arguing that this linear description of developing and delivering policy serves more to disorient strategic thinking and discussion than enlighten.

“There is not, and never will be, such a science [of evidence-based policy]” he says in a Financial Times article promoting the book.

Kay offers two distinct propositions: (1) evidence based policy does not happen; and (2) evidence based policy is not what should happen.

On his first point, anyone who has worked in government or social policy research knows that, whilst evidence-based policy has been the exception rather than the rule, things are changing (just have a look at the research pages of the National Policing Improvement Agency for example).

The second point shows Kay’s straw man for what it is – as many who apply research to policy have argued, the process is not about developing be evidence-based policy, but rather evidence informed policy. We are not so naive as to believe policy is or even should be developed on the basis of evidence alone – are we?

Lancet article calls for mandatory impact evaluation of public policy

Friday, March 12th, 2010

A group of highly respected scientists has published a paper in the Lancet calling for all governments to introduce mandatory use of research evidence to assess the likely effects of all public programmes before they are launched, and impact evaluations to monitor effectiveness post launch.

The paper cites as evidence of the political appetite for such a move the US government’s decision to ring-fence $1.1 billion for comparative research (including systematic reviews and clinical trials) as part of its $787 billion economic stimulus plan. Governments, they claim, have a political and ethical obligation to make effective decisions.

The paper goes on to cite the example of the Mexican Government which has passed legislation establishing a National Council for the Evaluation of Social Development Policy (CONEVAL). The Council is an independent public agency with a budget of around US$10 million. It not only conducts evaluations, but then publishes the results in the Official Journal of Evaluation.

Modelling the impact of government policy

Friday, February 19th, 2010

Dr Petra Meier, a Senior Lecturer in Public Health at the University of Sheffield, has received interesting press coverage following her testimony to the Scottish Parliament’s health committee.

Dr Meier has been modelling the likely impact of a minimum pricing policy on the drinking behaviour of Scots. In its coverage, the Scotsman newspaper reported Dr Meier as ‘admitting’ there was no evidence as to whether the policy would work as it had not been tried anywhere before. Much to the amazement of the Scotman’s reporter, Dr Meier described modelling as “like the weather forecast” which, as a Labour MSP (opposed to the policy) pointed out, “is notoriously unreliable”.

Modelling, as economists and operational researchers will tell you, is a valuable tool when it comes to the predicting the likely impact of social policy options. It seems there is an urgent need to explain to the press just what modelling is, and how it can make a valuable contribution to political decision making.

Comparative effectiveness research

Friday, February 19th, 2010

A symposium of the Fifth Annual Scientific Session of the Surgical Outcomes Club in Chicago, Illinois, in October of last year, was dedicated to the topic of comparative effectiveness research in surgery. The symposium served to clarify current thinking about what comparative effectiveness research is, particularly its objectives and methods.

The definition of comparative effectiveness research has evolved with the input of a variety of parties, including the Congressional Budget Office, Office of Management and Budget, and the White House. A working definition has been produced by the Federal Coordinating Council for Comparative Effectiveness Research that you can find here. Briefly, the definition talks about improving health outcomes by ‘developing and disseminating evidence-based information to patients, clinicians, and other decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances’.

Comparative effectiveness research is currently being used in some jurisdictions to make policy and coverage decisions. While policy making and coverage decisions are increasingly made on the basis of research evidence, the evidence on which decisions are made is typically very weak.

What Is Policy? And What Is Evidence-Based Policy-Making?

Friday, February 5th, 2010

I came across this on the intriguingly entitled Jack of Kent Blog, providing a skeptical and liberal view on public debates and legal issues.

The Jack of Kent Blog is written by Allen Green, a freelance legal and policy writer working in London.
This particular post explores definitions of policy, calling on such diverse sources as Samuel Johnson (“the art of government”) and RA Butler (“the art of the possible”), although I have always been lead to believe the latter was a quote from Otto Von Bismarck.

Either way, it’s an interesting blog, and well worth the visit to anyone interested in the art, or otherwise, of politics.

If you don’t know where you are going, any road will take you there

Friday, February 5th, 2010

If you don’t know where you are going, any road will take you there. If you don’t know where you are, you may already be there. Or you may not. – Alice in Wonderland

A great quote that comes from a presentation to the Prince Mahidol Award Conference in January by Dr Téa Collins. In her presentation entitled, ‘Measuring Health Systems Performance: Challenges and the need for more research’, Dr Collins set out the challenges that face researchers in estimating the impact of health systems, including the difficulty of establishing specific contribution of health systems to public health, the long-term nature of some clinical outcomes, and problems of collecting adequately robust data. You can download her slide presentation at the global forum for health website.

Evidence in education policy and practice

Friday, January 29th, 2010

I came across an article from Amanda Cooper and her colleagues in the Journal of Educational Change that looks at the issue of what they call ‘knowledge mobilization’ in the education field.

The paper gives an interesting overview of the issues surrounding the role of evidence in education, and considers what needs to happen to improve things. Whilst recognizing that policy will never be determined by evidence alone, the authors identify three priorities:

1. Organise and disseminate research output in ways that encourage practitioners to engage with it;
2. Encourage organisations that produce research to get more actively involved in knowledge mobilization;
3. Provide practitioners and the institutions in which they work with the resources that would enable them to find, share, understand and use research.

For those of you wanting to use the paper for purely personal research purposes, I found a pdf version here.

Reduce salt intake, save lives

Friday, January 29th, 2010

The emphasis of UK health policy has changed in recent years to focus much more sharply on prevention relative to cure. An article in this month’s New England Journal of Medicine suggests that public health interventions to reduce salt intake could deliver reductions in death and disease akin to smoking cessation campaigns.

In the article, Bibbins-Domingo and colleagues present evidence from the US to suggest that a public health intervention programme to reduce daily salt intake by 3 g (1200 mg of sodium) could reduce the annual number of new cases of coronary heart disease (CHD) by 60,000 to 120,000, stroke by 32,000 to 66,000, and myocardial infarction by 54,000 to 99,000 and reduce the annual number of deaths from any cause by 44,000 to 92,000.

A campaign to reduce salt intake could save 194,000 to 392,000 quality-adjusted life-years and $10 billion to $24 billion in health care costs annually. Even if the intervention reduced salt intake by just 1 g per day, the benefits would still be substantial and would warrant implementation.