Matrix Evidence

Although not widely reported in the press, we noted with interest more policy on the hoof from Nick Clegg at the 2011 LibDem September conference.

The idea of providing catch-up summer classes for children “most in need” prior to starting secondary school may have some intuitive appeal, but where similar schemes have been properly evaluated, questions have been raised as to their effectiveness. Is it a good idea to invest £50m of tax-payers money in a scheme with no proven efficacy?

One of the most well-known evaluation studies (amongst students of social policy at any rate) is the Cambridge Summerville Youth Study:

http://www.childtrends.org/lifecourse/programs/cambridge.htm

It was a community based program for children and adolescents in eastern Massachusetts in the 1950’s. Children in the program received a counsellor who visited with them and their family twice a month. These counselors are on call for problems that the family may have been having and referred children to a variety of different programs including tutoring, medical treatment, psychiatric treatment, summer camps, Boy Scouts, YMCA, or other community programs.

Unusually, the programme was robustly evaluated using a randomized controlled trail methodology. The results from a 30 year follow-up showed conclusively that the programme had no impacts on juvenile arrest rates measured by official or unofficial records. The programme also had no impacts on adult arrest rates. There were no differences between the two groups in the number of serious crimes committed, age at when a first crime was committed, age when first committing a serious crime, or age after no serious crime was committed. A larger proportion of criminals from the treatment group went on to commit additional crimes than their counterparts in the control group.

Other studies have come up with similar findings. Generally they conclude that programmes like this, unless they are of very high quality (well-staffed by trained professionals), tend to act as ‘bad behaviour academies’, where like-minded children get together to exchange ideas on how to be anti-social.

Without wanting to dismiss the idea of summer schools purely on the basis of a flawed programme implemented a generation ago, it would be nice to see a commitment to ring-fence part of the proposed £50m budget to some robust (RCT) evaluation.

Tags: Criminal justice, deprivation, education, evidence-based policy, randomised controlled trials
1 Comment »

I came across an article from Amanda Cooper and her colleagues in the Journal of Educational Change that looks at the issue of what they call ‘knowledge mobilization’ in the education field.

The paper gives an interesting overview of the issues surrounding the role of evidence in education, and considers what needs to happen to improve things. Whilst recognizing that policy will never be determined by evidence alone, the authors identify three priorities:

1. Organise and disseminate research output in ways that encourage practitioners to engage with it;
2. Encourage organisations that produce research to get more actively involved in knowledge mobilization;
3. Provide practitioners and the institutions in which they work with the resources that would enable them to find, share, understand and use research.

For those of you wanting to use the paper for purely personal research purposes, I found a pdf version here.

Tags: education, evidence-based policy
2 Comments »

William G. Wraga, a professor in the University of Georgia College of Education’s Department of Lifelong Education, Administration, and Policy has recently published an interesting article on the evidence surrounding charter schools.

The original intent of charter schools, to increase the professional autonomy of teachers so they could explore innovative ways to educate children and youth, has given way, Professor Wraga claims, to other agendas that have been grafted onto the movement.

Increasingly, charter school policies have been influenced by market ideology that treats the movement as a vehicle for privatizing public schools.

However, Professor Wraga points out that available evidence from the US suggests, in practice, charter school “innovations” too often occur more on the management side than on the educational side of schooling. Overall, he asserts, charter school students perform no better than non-charter public school students.

To improve the education of all students, policymakers should focus not on the governance structures of schools, but on the improvement of curriculum and instruction in classrooms.

If our goal is to improve education for all students, the bulk of the evidence suggests that market-based education reforms, including charter schools, are not going to help us achieve it.

The best policy for improving public schools is to invest directly in reforms that have been proven to work.
Those reforms should focus not on the business side of schools, but on implementing classroom practices that have been demonstrated to improve curriculum, instruction and assessment—and ultimately student learning.

Tags: education, evidence-based policy
1 Comment »

Andy Smarick, an education researcher at The Thomas B. Fordham Institute in the US has published evidence to suggest that increasing investment in failing schools doesn’t always lead to improvements. Smarick cites data from an evaluation of the ‘No Child Left Behind’ initiative. According to Smarick, a report from the US Department of Education suggests only around 19% of schools restructured under the programme were able to demonstrate sufficient improvement to leave the scheme within a two year period. And it’s not just schools; Smarick suggests his findings are consistent with evidence concerning the impact of schemes like Total Quality Management (TQM) and Business Process Reengineering (BPR) on private business, successful in improving performance in around only one in three organisations.

Tags: education
No Comments »