Throughout October and November, the Independent Online has been partnering with the Institute of Ideas Battle of Ideas festival to present a series of guest blogs from festival contributors. One of the contributors, Ben Pile, posted an intriguing article entitled Against evidence-based policy making. In it, he engages in one of those slightly annoying reductionist arguments where he rails against a position that no one apart from himself would support. However, he finally comes around to concluding something many would support, that an ‘….over-emphasis [my italics] on evidence in policy making is corrosive to a public debate about the values which inform the interpretation of evidence’. An interesting article and even more interesting set of responses can be found here
Evidence Corner - Posts Tagged ‘evidence-based policy’
‘Independent’ Battle of Ideas Blog
Monday, December 5th, 2011
Using evidence to set healthcare priorities
Friday, September 23rd, 2011
The Nuffield Trust and the Health Services Management Centre (HSMC) at the University of Birmingham have carried out a national research study into how primary care trusts (PCTs) have gone about setting priorities for health care spending. Setting priorities in health maps the activities taking place across the NHS and explores the effectiveness of these practices within specific local contexts.
http://www.nuffieldtrust.org.uk/sites/files/nuffield/setting-priorities-in-health-research-report-sep11_0.pdf
Evaluating summer camps for children in need
Friday, September 23rd, 2011
Although not widely reported in the press, we noted with interest more policy on the hoof from Nick Clegg at the 2011 LibDem September conference.
The idea of providing catch-up summer classes for children “most in need” prior to starting secondary school may have some intuitive appeal, but where similar schemes have been properly evaluated, questions have been raised as to their effectiveness. Is it a good idea to invest £50m of tax-payers money in a scheme with no proven efficacy?
One of the most well-known evaluation studies (amongst students of social policy at any rate) is the Cambridge Summerville Youth Study:
http://www.childtrends.org/lifecourse/programs/cambridge.htm
It was a community based program for children and adolescents in eastern Massachusetts in the 1950’s. Children in the program received a counsellor who visited with them and their family twice a month. These counselors are on call for problems that the family may have been having and referred children to a variety of different programs including tutoring, medical treatment, psychiatric treatment, summer camps, Boy Scouts, YMCA, or other community programs.
Unusually, the programme was robustly evaluated using a randomized controlled trail methodology. The results from a 30 year follow-up showed conclusively that the programme had no impacts on juvenile arrest rates measured by official or unofficial records. The programme also had no impacts on adult arrest rates. There were no differences between the two groups in the number of serious crimes committed, age at when a first crime was committed, age when first committing a serious crime, or age after no serious crime was committed. A larger proportion of criminals from the treatment group went on to commit additional crimes than their counterparts in the control group.
Other studies have come up with similar findings. Generally they conclude that programmes like this, unless they are of very high quality (well-staffed by trained professionals), tend to act as ‘bad behaviour academies’, where like-minded children get together to exchange ideas on how to be anti-social.
Without wanting to dismiss the idea of summer schools purely on the basis of a flawed programme implemented a generation ago, it would be nice to see a commitment to ring-fence part of the proposed £50m budget to some robust (RCT) evaluation.
Launch of rigorous evidence newsletter
Friday, February 4th, 2011
With the strap line ‘Distinguishing Effective Evidence-Based Programs from Everything Else’, the US Coalition for Evidence-Based Policy has launched its monthly newsletter. Circulated as an e-newsletter, it summarizes key findings from high quality (the focus is on well-conducted RCTs) evaluations across all areas of social policy. The Coalition follows guidelines issued by the US National Academies on evaluation designs.
The newsletter provides results from the Coalition systematically monitoring the literature to identify all new RCTs – and other rigorous nonrandomized evaluations – published or posted on-line across all areas of social policy. It includes (i) summaries of the few findings identified by an expert panel as meeting the highest (Congressional “Top Tier” or near Top Tier) evidence standards; as well as (ii) summaries of promising findings that are not yet ready for Top Tier consideration (e.g., due to only short-term follow-up). The Coalition views these latter findings – promising but not conclusive – as valuable for identifying programmes that merit testing in more definitive RCTs.
Looks like another valuable resource for the evidence-based policy community?
Beyond evidence-based policy: Outcome-based government
Friday, January 28th, 2011
The Centre for Social Justice this week published a report entitled Outcome-based Government: How to improve spending decisions across government.
In what should be essential reading for anyone in the business of producing evidence and analysis for policy, the report sets out how both local and national government needs to up their game when it comes to applying the principles of evidence-based policy. As an example of the failure of government to use evidence and analysis adequately, it quotes the Public Accounts Committee as observing: ‘The Government spends £1.2 billion a year on measures aimed at tackling problem drug use, yet does not know what overall effect this spending is having.’
The report provides an excellent analysis of why evidence has not been applied consistently to the development and evaluation of policy. One observation has particular resonance for anyone who has worked in government research:
‘The late involvement of analysts in policy development led one civil servant to describe the process as “policy-led evidence making”, with the methodology often used to support ministerial initiatives rather than appraise policy options.’
As with any good analysis, the report offers a menu of practical solutions to the problem. Top of the list comes the need for government to articulate a specific set of outcomes that define its priorities. It’s all very reminiscent of the line argued in this blog of the value for local and national government of having a clear theory of change for policy interventions. Having robust empirical evidence for links between outputs and outcomes is an essential pre-requisite to effective monitoring and evaluation.
Whilst the report recognises the value of methods described in the Treasury’s Green Book, it notes that in too many cases, the guidance is not followed consistently. As anyone who has discussed the ROAMEF model with policy makers will know, in cases where its principles are applied, it is often too late after key policy decisions have been made.
Among the twenty, very practicable, recommendations the report makes, perhaps the most standout (for a right of centre think tank) is the establishment of a new, independent body to scrutinise the extent to which policy decisions in local and national government are adequately informed by robust analysis. The proposed new body, The Office of Spending Effectiveness (OSE), would be responsible for setting standards of decision making, ensuring policy was adequately informed by evidence-based business cases, and evaluated through the collection and analysis of robust cost and outcome data. Such a scrutiny function could potentially make great strides towards delivering evidence-based policy.
Whilst you may not agree with all of the proposals made in the report, it certainly reads like a refreshing new approach to the perennial problem of how to get evidence consistently integrated into the process of government, both locally and nationally. A valuable contribution to the debate?
Evidence and employment policy
Friday, January 21st, 2011
Picking up the theme of evidence in the area of employment policy from the previous post (see below), the impressive ‘twenty-first floor’ site explores the evidence underpinning a Coalition announcement of changes to employment protection legislation. The proposals aim to extend the period after starting work within which employees cannot make claims of unfair dismissal from one to two years, and to require workers to post a fee when lodging an employment tribunal claim.
In essence, the theory of change underpinning this move is that giving employers greater flexibility to dismiss employees will stimulate the jobs market.
So what is the evidence? Two sources are relevant: First, the CBI reported findings from an employer survey showing that 46% felt employment tribunals were ineffective, and 50% claimed to have seen a rise in the number of vexatious claims. Second, a Trade Union Congress sponsored report, ‘Flexible with the Truth’, looks in detail at the relationship between labour market flexibility (LMF) and economic performance. It concludes the evidence that relaxing employment protection laws stimulates economic performance is equivocal at best. To quote: ‘…it is unlikely that a substantial reduction in EP …… would deliver drastically enhanced labour market performance’.
Author of the 21st Floor article Keir Liddle declines to take a definitive position on the grounds of insufficient evidence. How would you call it?
evidence about evidence-based policy
Monday, January 17th, 2011
The Centre for Science and Policy, University of Cambridge is launching a consultation to identify the key research questions on the relationship between science and policy.
The work is predicated on the assumption that we know very little about the relationship between the production of evidence and the making of policy. To date, research on integrating evidence into the policy making process has focussed either on the production of evidence, or on the mechanisms underlying the development of policy.
The Centre is taking up the challenge of building a research community whose remit will be to look closely at how scientific advice for policy audiences is produced. It wants to build alliances across academic disciplines, and also, crucially, with policy practitioners who work in this area, and also with those experts who have practical knowledge of what it is like to serve as expert advisers.
In the meantime, I came across this brief parliamentary interchange, care of the donpaskini blog:
Nic Dakin: To ask the Secretary of State for Work and Pensions what evidence he used to determine that planned changes to housing benefit for those out of work for over 12 months will increase employment levels.
Steve Webb: We did not make any specific assumptions about the impact on employment levels of this measure. Research shows that the reasons for long term unemployment are complex. However we believe reducing housing benefit after 12 months will provide an additional financial incentive for jobseekers to take up work.
Evidently some work to be done over at DWP on developing evidence-based policy!
Big society and healthcare: evidence for the impact of public involvement
Friday, January 7th, 2011
Big Society has become a signature policy of the current administration. While simple definitions are elusive, David Cameron has variously described it in terms of liberalism, empowerment, freedom and responsibility. In the context of delivering health and social care, an Institute for Government seminar concluded that health professionals will have to accept the public’s growing role in determining their own service needs.
But what is the evidence for greater public involvement improving services? Two recent papers provide some interesting insights.
Martin Marshall and Vin McLoughlin of the Health Foundation claim the available evidence does not unequivocally support the assumption that patient choice is a significant driver of quality improvements. Because patients typically do not behave like rational consumers when it comes to their healthcare, current mechanisms for providing them with information on care quality is not proving to be a strong lever for change. Marshall and McLoghlin suggest that changing the way information is presented to the public could change relationships between patients and providers for the better, but is unlikely to lead to improvements in quality of care in the short term.
A systematic review from the RAND Corporation found that while it may have some intrinsic value, there is little evidence to suggest public involvement in healthcare activities has had any positive impact on decision-making or policy. The report suggests that greater public involvement in health decisions may even threaten equitable access to services. It concludes that we in fact know too little about how people become involved as active service users, how the healthcare system influences patient expectations, and the impact of social and political values.
While the rhetoric may be popular, it would appear there is still work to be done in building evidence of effective practice when it comes to involving the public in policy and practice in the healthcare field.
Using anecdotes in presenting evidence
Friday, September 24th, 2010
A wise sage once said, ‘the plural of anecdote is not evidence’. Many who work producing evidence for policy will have faced the frustration of having decision makers weighting stories more highly than good quality empirical evidence.
However, a blog I came across this week makes a very reasonable point about how empirical scientists can sometimes use anecdotes very effectively to illustrate points supported by robust data. I find myself agreeing with the author – effective evidence for policy need not always avoid anecdotes. We should recognize their limitations of course, but at the same time recognize their power to reinforce, for certain audiences, messages that have been drawn from robust empirical evidence. It’s by no means an easy feat to get the right mix, and in some instances, for some audiences, the use of anecdotes is inappropriate. But as any good communicator will tell you, stories can be a very useful tool in getting messages across. As scientists working in the evidence for policy arena, we ignore that fact at our peril.
Research for Recovery:A Review of the Drugs Evidence Base
Friday, September 10th, 2010
The Scottish Government, on behalf of the National Drugs Evidence Group, has published a review of the drugs evidence base. The review aims was to highlight where the evidence base is already strong, what the evidence tells us and where the knowledge gaps are in relation to Scotland’s National Drugs Strategy, The Road to Recovery: A New Approach to Tackling Scotland’s Drug Problem.
The review looks at both the published evidence and the policy context in which the strategy sits. It provides the link between the evidence around addictions and the wider social, health and economic context in which recovery occurs. The review set out to assess what we know about ‘what works’ in drugs recovery and identify the core questions that need to be answered when assessing the effectiveness and impact of The Road to Recovery.