Rosie McLeod: "When your peers only report successes, it’s hard to be the odd one out"

Why should an organization want to disseminate the results from an impact assessment of a programme which has not been a success?

Because it’s important for other organizations to hear about what hasn’t worked. Otherwise it’s likely to be repeated elsewhere, which is a waste of effort and unfair on beneficiaries. And if the organization had good reason to believe the programme should work, there’s no shame in finding out that it doesn’t – how could they have known? It shows they are trailblazers, and that they are advancing the field by testing new ideas.

Why are organizations not reporting unexpected results more often?

Generally because nobody else does! When your peers only report successes, it’s hard to be the odd one out, coming clean about the messy reality of mixed results. And there’s a fear of putting people off – it’s not an easy thing to do. Many feel that to maintain the confidence of your funders, your staff and your beneficiaries, you need to demonstrate success, and only share the good news stories rather than the lessons.

How can an organization learn from a bad experience?

By properly analysing what happened and why. Be prepared to diagnose the causes. Was it an issue with the programme design, or the way it was implemented? Were the unknowns obvious from the start? Crucially, staff need to be engaged in that review, so the organization gets the best insights into why things didn’t go to plan, and avoid blame wherever possible. Nobody is necessarily at fault, and if staff are defensive you will get nowhere.

How should an impact assessment be in order to permit the organization learn from unexpected results?

Plan an impact assessment that aims to understand what works or doesn’t, and allows you to pick up and make changes as you go. If you anticipate the areas that are most risky, you are better prepared for surprises that may come. Staff need to be encouraged to demonstrate improvement, not just success, and set themselves learning goals, not just delivery goals. Programmes also need to use defined indicators so benchmarks and progress are clear.

Can all impact assessment methods lead to get negative results?

If you ask the right questions, your results will tell you something new and useful, regardless of whether they are ‘positive’ or ‘negative’ in terms of achieving outcomes. ‘Does the programme achieve X outcome’ can always yield a negative result, and we always want that straight answer. But alongside that, you need methods that capture the programme’s mechanisms, the variation among users, the other factors influencing whether it worked. Those things can help you adapt the programme or suggest some issues a new programme would need to take account of.

For some third sector leaders, impact assessment is seen as a tool to disseminate success. What do you think about it?

Well everyone wants to look good, but how about focusing on being good too? Great organisations that care about impact will use impact assessment to guide their decisions, which means they need balanced information that tells them things they don’t already know. Funders have a really important role to play in reassuring organisations that the aim is to improve, not just prove. It’s funders who often set reporting requirements and create the incentives or barriers to sharing difficult results.    

How can an organization report unexpected results without compromising its reputation or its funding?

If you have tried to be open about the risk and unknowns from the beginning, unexpected results are less of a surprise and don’t reflect badly on you. Findings can be framed with positives. You can demonstrate your ability to learn by analyzing results and suggesting new approaches. You can share with people what changes you are making as a result, to address the issue and continually improve.

Keep an open line of communication with your funders – that’s what maintains trust and confidence. Don’t hide results, discuss them as early as possible and explain what’s going on. More generally, look for funders who share your risk appetite. Some funders may not be interested in supporting risky projects that may not work, while some charities are less inclined to take risks with programmes with several unknowns. Charities and funders need to recognise their risk appetites, so they can find like-minded collaborators and manage risk.

Could you tell us some good practices of organizations sharing unexpected results and learning from them?

Jamie Oliver Foundation’s Fifteen, when it started out. In its first social report, it admitted the support it offered the troubled young people it took on as chefs-in-training was not as systematic and comprehensive as it could have been. The team decided to make several changes to their programme. As a result, the retention rate for trainees increased to 79% and they could say they felt they were improving all the time.

The Hewlett Foundation is so aware of the tendency for failures to be concealed that it has an annual award for the staff member who made its 'worst grant'.

The aptly named Admitting Failure is "an open space for development professionals who recognise that the only "bad" failure is one that's repeated".