Matrix Evidence

Although not widely reported in the press, we noted with interest more policy on the hoof from Nick Clegg at the 2011 LibDem September conference.

The idea of providing catch-up summer classes for children “most in need” prior to starting secondary school may have some intuitive appeal, but where similar schemes have been properly evaluated, questions have been raised as to their effectiveness. Is it a good idea to invest £50m of tax-payers money in a scheme with no proven efficacy?

One of the most well-known evaluation studies (amongst students of social policy at any rate) is the Cambridge Summerville Youth Study:

http://www.childtrends.org/lifecourse/programs/cambridge.htm

It was a community based program for children and adolescents in eastern Massachusetts in the 1950’s. Children in the program received a counsellor who visited with them and their family twice a month. These counselors are on call for problems that the family may have been having and referred children to a variety of different programs including tutoring, medical treatment, psychiatric treatment, summer camps, Boy Scouts, YMCA, or other community programs.

Unusually, the programme was robustly evaluated using a randomized controlled trail methodology. The results from a 30 year follow-up showed conclusively that the programme had no impacts on juvenile arrest rates measured by official or unofficial records. The programme also had no impacts on adult arrest rates. There were no differences between the two groups in the number of serious crimes committed, age at when a first crime was committed, age when first committing a serious crime, or age after no serious crime was committed. A larger proportion of criminals from the treatment group went on to commit additional crimes than their counterparts in the control group.

Other studies have come up with similar findings. Generally they conclude that programmes like this, unless they are of very high quality (well-staffed by trained professionals), tend to act as ‘bad behaviour academies’, where like-minded children get together to exchange ideas on how to be anti-social.

Without wanting to dismiss the idea of summer schools purely on the basis of a flawed programme implemented a generation ago, it would be nice to see a commitment to ring-fence part of the proposed £50m budget to some robust (RCT) evaluation.

Tags: Criminal justice, deprivation, education, evidence-based policy, randomised controlled trials
1 Comment »

With the strap line ‘Distinguishing Effective Evidence-Based Programs from Everything Else’, the US Coalition for Evidence-Based Policy has launched its monthly newsletter. Circulated as an e-newsletter, it summarizes key findings from high quality (the focus is on well-conducted RCTs) evaluations across all areas of social policy. The Coalition follows guidelines issued by the US National Academies on evaluation designs.

The newsletter provides results from the Coalition systematically monitoring the literature to identify all new RCTs – and other rigorous nonrandomized evaluations – published or posted on-line across all areas of social policy. It includes (i) summaries of the few findings identified by an expert panel as meeting the highest (Congressional “Top Tier” or near Top Tier) evidence standards; as well as (ii) summaries of promising findings that are not yet ready for Top Tier consideration (e.g., due to only short-term follow-up). The Coalition views these latter findings – promising but not conclusive – as valuable for identifying programmes that merit testing in more definitive RCTs.

Looks like another valuable resource for the evidence-based policy community?

Tags: evidence-based policy, randomised controlled trials, USA
3 Comments »

The Australian Productivity Commission organised a roundtable discussion on the issue of strengthening evidence-based policy in 2009. Participants included government officials, academics, consultants and representatives of non-government organisations. Keynote addresses were presented by Dr Ron Haskins, Senior Fellow of the Brookings Institution, and Professor Jeffrey Smith of the University of Michigan.

The roundtable discussed the principles of the evidence-based policy movement and reviewed how well Australian use of evidence conformed to best practice. It then considered how to improve the availability of quality evidence, and reviewed possible institutional developments to embed good use of evidence more firmly into policy-making.

The roundtable proceedings have now been published. They include papers by the speakers and a summary of the key points covered in the discussion sessions, and a series of background papers prepared by Commission staff and provided to roundtable participants.

I was particularly drawn by a paper entitled ‘Evidence-based policy: summon the randomistas’ by Andrew Leigh of the Research School of Economics at the Australian National University

Tags: evidence-based policy, randomised controlled trials, systematic reviews
No Comments »

For anyone interested in evaluation of social policy, the US Government Accountability Office (GAO) is an extremely useful source of robust evidence.

The GAO was recently asked to take a look at an initiative delivered by the Coalition for Evidence-Based Policy. The Top Tier evidence initiative involved the CEP reviewing the evidence associated with 63 different interventions; of those 63, the CEP rated 6 as top tier, defined in Congressional legislation as ‘well-designed randomized controlled trials [showing] sizeable, sustained effects on important ….outcomes’.

The GAO concluded that RCTs are the best way of providing robust evidence of intervention effectiveness. However, they also noted that RCTs are often difficult to conduct in the context of social policy evaluation. For that reason, reliance on only evidence from RCTs would severely limit the evidence available to policy makers. Therefore the GAO concluded that several rigorous alternatives should be considered including quasi-experimental designs and statistical analysis of observational data.

Other important points raised by the GAO included: (i) the decision to adopt an intervention will be informed not just by evidence of effectiveness, but also by data on costs and benefits, and suitability to local communities, and (ii) improved evaluation would help identify effective interventions.

Tags: Add new tag, evidence debate defence, randomised controlled trials
1 Comment »