Matrix Evidence

The Centre for Social Justice this week published a report entitled Outcome-based Government: How to improve spending decisions across government.

In what should be essential reading for anyone in the business of producing evidence and analysis for policy, the report sets out how both local and national government needs to up their game when it comes to applying the principles of evidence-based policy. As an example of the failure of government to use evidence and analysis adequately, it quotes the Public Accounts Committee as observing: ‘The Government spends £1.2 billion a year on measures aimed at tackling problem drug use, yet does not know what overall effect this spending is having.’

The report provides an excellent analysis of why evidence has not been applied consistently to the development and evaluation of policy. One observation has particular resonance for anyone who has worked in government research:

‘The late involvement of analysts in policy development led one civil servant to describe the process as “policy-led evidence making”, with the methodology often used to support ministerial initiatives rather than appraise policy options.’

As with any good analysis, the report offers a menu of practical solutions to the problem. Top of the list comes the need for government to articulate a specific set of outcomes that define its priorities. It’s all very reminiscent of the line argued in this blog of the value for local and national government of having a clear theory of change for policy interventions. Having robust empirical evidence for links between outputs and outcomes is an essential pre-requisite to effective monitoring and evaluation.

Whilst the report recognises the value of methods described in the Treasury’s Green Book, it notes that in too many cases, the guidance is not followed consistently. As anyone who has discussed the ROAMEF model with policy makers will know, in cases where its principles are applied, it is often too late after key policy decisions have been made.

Among the twenty, very practicable, recommendations the report makes, perhaps the most standout (for a right of centre think tank) is the establishment of a new, independent body to scrutinise the extent to which policy decisions in local and national government are adequately informed by robust analysis. The proposed new body, The Office of Spending Effectiveness (OSE), would be responsible for setting standards of decision making, ensuring policy was adequately informed by evidence-based business cases, and evaluated through the collection and analysis of robust cost and outcome data. Such a scrutiny function could potentially make great strides towards delivering evidence-based policy.

Whilst you may not agree with all of the proposals made in the report, it certainly reads like a refreshing new approach to the perennial problem of how to get evidence consistently integrated into the process of government, both locally and nationally. A valuable contribution to the debate?

Tags: economics, evidence-based policy, Local Authorities
5 Comments »

In a book to be published this month, “Obliquity – how our goals are best pursued indirectly” British economist John Kay looks at the reasoning behind the notion of evidence-based policy and finds it wanting.

Kay defines evidence-based policy making as a process of determining the goals of a policy intervention, and then looking for evidence to determine the most effective means of delivering those goals. This, he claims, is not a good description of how policy decisions are made in reality, or could ever be.

Having set up his straw man, Kay goes on to knock it down, arguing that this linear description of developing and delivering policy serves more to disorient strategic thinking and discussion than enlighten.

“There is not, and never will be, such a science [of evidence-based policy]” he says in a Financial Times article promoting the book.

Kay offers two distinct propositions: (1) evidence based policy does not happen; and (2) evidence based policy is not what should happen.

On his first point, anyone who has worked in government or social policy research knows that, whilst evidence-based policy has been the exception rather than the rule, things are changing (just have a look at the research pages of the National Policing Improvement Agency for example).

The second point shows Kay’s straw man for what it is – as many who apply research to policy have argued, the process is not about developing be evidence-based policy, but rather evidence informed policy. We are not so naive as to believe policy is or even should be developed on the basis of evidence alone – are we?

Tags: economics, evidence-based policy
3 Comments »

Dr Petra Meier, a Senior Lecturer in Public Health at the University of Sheffield, has received interesting press coverage following her testimony to the Scottish Parliament’s health committee.

Dr Meier has been modelling the likely impact of a minimum pricing policy on the drinking behaviour of Scots. In its coverage, the Scotsman newspaper reported Dr Meier as ‘admitting’ there was no evidence as to whether the policy would work as it had not been tried anywhere before. Much to the amazement of the Scotman’s reporter, Dr Meier described modelling as “like the weather forecast” which, as a Labour MSP (opposed to the policy) pointed out, “is notoriously unreliable”.

Modelling, as economists and operational researchers will tell you, is a valuable tool when it comes to the predicting the likely impact of social policy options. It seems there is an urgent need to explain to the press just what modelling is, and how it can make a valuable contribution to political decision making.

Tags: economics, evidence-based policy, operational research
No Comments »

An increasing part of the review work we do incorporates economic evaluations. People want to know, not just what works, but what the relative costs and benefits are likely to be when implementing effective interventions, medical or social.

A recent paper published by Rob Anderson, from Exeter University, looks at how useful systematic reviews of economic evaluations are in the context of health care. He argues that synthesising results across lots of different economic evaluations is risky for three reasons: (1) economic data is very context dependent, so it’s dangerous to generalise across settings; (2) assuming results can be generalised is crucial if the findings are going to inform policy decisions; and (3) decision analytic modelling, one of the main techniques economists use, is itself a synthesis tool, so doing additional systematic reviewing may be redundant.

The paper does not claim synthesising economic data is not useful or informative; it’s just that there may be more effective ways of doing it than systematic reviews.

Tags: economics, meta-analysis, systematic reviews
8 Comments »