Statistics need the stories behind them

Measuring the impact of a service isn’t simply about statistics – you need to highlight the human side. Your funding may well depend on it, says Tim Crabbe

Last month the youth sector was accused of failing to explain the difference it makes. This indictment came from the chair of the education select committee inquiry into young people’s services who said the lack of good quality evidence was weakening the sector’s case for government funding.

The funding tap is already running dry. According to recent research, one in four of England’s youth services face severe cuts of between 21-30%.

But amid the doom and gloom of cuts there recently came some good news. Positive Futures – a programme to help prevent young people becoming involved in crime – will continue to receive £10m of Home Office funding over the next two years.

So what has Positive Futures done differently? At such a difficult time, why would government make a significant investment in this particular programme? The answer goes back to hard evidence.

Young people take part in the evaluation of Positive Futures.

This isn’t just about statistics. Yes, latest figures from the 91 Positive Futures projects around the country do show improved engagement among 70% of participants with over 10,000 gaining qualifications and almost 30,000 achieving other positive outcomes including employment, volunteering and a return to education.

But numbers are married up with the emotional and social benefits of the programme – recorded through a unique monitoring system that charts the progress of every young person. Project managers record how youngsters interact with peers and staff, the contribution they make to a session and their individual achievements. Case studies, filmed interviews, photos and artwork all help to tell the young people’s stories.

Take Daafi. He’s been involved with Coventry Positive Futures for the past six months. At 14 he was caught shoplifting. His attitude at school was described as ‘horrific’ and he was removed from mainstream lessons. Staff at Coventry Positive Futures developed a learning timetable for Daafi, overseen by a peer mentor rather than a teacher. He would study in the informal atmosphere of Coventry’s Hillfields Centre and in the evening he would attend a sports session. Over three months Daafi’s behaviour improved dramatically and he was given access to mainstream education again.

Coventry Positive Futures has hundreds of stories like Daafi’s to tell. Staff collect demographic data about young people and track their achievements: did they enroll on a smoking cessation programme, complete their basketball coaching course or design a project poster? Project workers track the programme’s impact through one-to-one interviews with young people and comments on the project’s Facebook wall. Young people keep short diaries and take photos of their peers winning awards.

The traditional approach to evaluation is very different to this. Ticking boxes and completing milestones has always been viewed by front line agencies as a bureaucratic burden, a distraction from the work itself and something ‘done’ to them to justify government policies. Those at the receiving end often don’t see how narrow sets of key performance indicators truly represent the outcomes of their work.

Substance, the social research co-operative that is currently responsible for the evaluation of Positive Futures in partnership with Sheffield Hallam University, points to a new model of impact monitoring, called Views, which is being launched this month and that it hopes will replace the traditional output focused approach.

This new way of proving impact is rooted in the day-to-day experiences of frontline workers and service-users, relying heavily on case studies, photos, videos and interviews as well as more traditional data. Embedding data-capture mechanisms in routine processes leads to a higher quality of impact information that can be mapped against outcome frameworks. In turn, at a time of pressure on the public finances, this allows commissioners to respond in a more strategic and thoughtful fashion that rewards delivery that is demonstrating effective results as well as flagging up those services that might need to be adjusted or cut.

• The new Views impact monitoring system is currently being piloted by four Positive Futures projects.
  Visit www.views.coop or find out more at Substance’s annual conference on 4 March http://www.anewballgame.substance.coop/