Back in May 2013, some data was published by Giving Evidence in the UK and GiveWell from the States, which they claimed showed that the most effective charities had high levels of admin costs. In December, the Guardian Voluntary Sector Network said it was one of the most read blogs of the year and Giving Evidence headlined a piece in their newsletter ‘Good charities have admin costs’. In their original press release they said:

“It’s unarguably wrong of donors such as Gina Miller to suggest that admin costs be capped. The data indicate that such caps would nudge donors towards choosing weaker charities, at untold cost to their beneficiaries.”

The debate on admin costs is important. Our research shows that donors are very keen to make sure their money is well spent and don’t like admin costs. Equally, we know that many within the sector don’t like the fact that charities are often judged by the size of their admin costs, a sentiment we would share. However, it is important that we base our case against using admin costs on solid evidence. So with a bit of spare time over the Xmas break, we thought we would examine the data behind the claim that effective charities spend more on admin. This is what we found.

The data for the original study came from charities that GiveWell has analysed in the US. The argument is that the charities that GiveWell ranks highly (i.e. the 11.5% group below) have a higher level of admin costs than those they said were not so good or didn’t rank (i.e. the 10.8% below group).

In a nutshell, we found:

• There is no statistical significant data that shows that high performing charities have higher admin costs – not even close
• This is because the sample size is way too small (only six in one group) and the differences in admin levels too small
• The charities studied are from the US and mostly work in international development
• Admin costs have no definition in UK charity accounting – they are the amount that remains when everything else is accounted for
• Even if the data were valid, we wouldn’t know whether high admin levels are the cause of effectiveness or the effect of something else, such as an effective CEO

Now in more detail…

The number of charities in the sample is simply too small

To recap, the headline said that effective charities spent 11.5% on admin, whereas ineffective charities spent 10.8% on admin for 2010-11. There were only six (yes just six) charities whose data was included in the 11.5% figure and just over 20 for the 10.8% figure. This isn’t anywhere close to being statistically significant (the data has a p value of 0.89 for those of you who like those things).

Equally important, within the groups there was a huge range of values. For example, the lowest level of admin costs in the ‘high-performing’ 11.5% group was 2% (so how does that fit with the argument that ‘good charities have admin costs’ - we really don’t understand) and the highest 22%.

There was another sample group from 2008-9 which had slightly different headlines. Here, the high performing group had admin levels of 10.2% and the lower performing group had admin levels of 9.4%. The numbers in that sample still aren’t high. The high performing group had 37 organisations in it. So, better than the 2010-11 group but still not strong enough to call the case “unarguable”. One analysis on the Freakonomics website in support of not using ‘overheads’ as a way of judging charities found no statistical significance in these numbers (more of this below).

One brain-scratching conundrum that eagle-eyed numbers geeks may have spotted is that the high performers in 2008-9 had a lower admin level (10.2%) than low performers in 2010-11 (10.8%). So, is it the relative level of admin that is alleged to make the difference or the absolute level? If it’s the latter, then 2010-11’s low performers would have been just fine in 2008-9. Or they would have ruined the arguments that the authors wanted to make.

Here is the authors’ helpful further breakdown of the admin levels for 2008-9:

So if we remove the six gold and silver organisations from the 2008-9 data, the difference between the high performers and the low performers is just 0.25%. Is this difference statistically significant? No, it isn’t. Whether measured with or without the gold and silver performers, there is no meaningful difference between the high performers and the rest in terms of admin costs.

The sample for the study is largely international charities and the rest are US-based.

The charities in the sample are three-quarters ‘international’. The rest are ‘United States’. It’s hard to see how a sample like this is relevant or applicable to medical, disability, social welfare or environmental charities, even if the basic data was valid.

The point about studying US charities (international or other) is that we don’t know whether US financial accounting can cross the Atlantic. One reason to be sceptical about US data is that in the UK, there is no definition of admin under the Charity Commission’s SORP. Admin is general classified as the expenditure that is left when everything else is taken out (mostly charitable activities and fundraising). So, different charities will treat the cost of buildings, evaluation costs, secretarial costs and the like very differently, meaning it’s very hard to compare admin costs even within the UK. Even if data like this is relevant in the States, it’s hard to see how it has much value this side of the Atlantic.

More generally, the sample is important in order for a study like this to be useful. This is because we need to know that the sample of charities analysed has a clear rationale, say by income, e.g. 10 organisations in each of 10 income bands, or from 10 different fields of work. At the moment, it’s very hard for any other organisation to replicate the analysis. At the very least, a key thing we need to know is how those organisations analysed by GiveWell are chosen. Are they chosen at random, do they pay to be analysed, or does GiveWell analyse those that funders want to analyse perhaps? This approach to picking a sample for research would never withstand scrutiny in any medical trial.

What do other analysts say about this data?

A 2011 article by Dean Karlan on the website Freakonomics entitled ‘Why ranking charities by administration expenses is a bad idea’ (a sentiment with which we wholly agree) analyses the GiveWell data. His analysis shows that there is no difference in the admin costs of the high and low performers, with a p value of 0.35 (in stats speak, this figure would need to be closer to 0.05 for it to be statistically significant). Remember this is the same data that Giving Evidence says makes the case “unarguable”. Even more interesting is that his analysis also covers fundraising costs, which he says show a statistically significant difference for high and low performers. Confused as to how the same set of data can lead to such a different conclusion? Us too.

Where are the fundraising costs?

So this leads us to another point. Adding together the programme costs and the admin costs for both high and low performing charities gives around 92-93% of total income. This implies that fundraising costs are around 7% (though we have no evidence for that, other than the income that is left unaccounted for). If fundraising costs are around 7%, then that is low compared to the UK, where they might be more typically around 20% or higher. So where are the fundraising costs in this data?

If better charities had high admin costs – would it be cause or effect?

Let’s suppose for a moment that the data is accurate and the admin difference is statistically significant. The question would then be whether the difference in admin costs is the cause of greater effectiveness or that high admin costs and effectiveness are just the results of another factor, such as an energetic, focused and strategic CEO. Just because there is a link between admin costs and effectiveness doesn’t automatically mean that high admin costs cause effectiveness. They could both be impacted on by something else altogether. Put another way, any charity which increased its admin costs in the sole hope of being more effective would be sorely disappointed.

What’s our conclusion?

We believe that there is no evidence that stands up to scrutiny about the level of admin costs and the performance of charities. However, the basic premise that the level of admin costs is a poor way to judge charities is one we would absolutely agree with. One problem is that a charity’s admin costs may tell you more about the skill and financial wizardry of a finance director than anything else.

All our research with the public suggests that they do care deeply about levels of admin costs because they see them as synonymous with waste. Our real challenge then is to give donors an alternative to admin costs as a way of judging charities. In this sense, what GiveWell are doing in the States with assessing charities is welcome. However, that does not justify using those assessments to make exaggerated claims about admin levels based on weak or inconclusive data.

Oh and remember Gina – it’s a free world. You’re a donor. You can argue what you feel. It’s the job of charities to change your mind and if necessary find some hard data to prove our case.

A nod of the head? A furrowing of the brow? Leave us a comment below.