Industries / Government, education and community / Nonprofits
Measuring results
Evaluators, researchers and raters try to find out whether a charity's programs work, what they cost per result, and how charities compare.
6
Milestones achieved, from 2001 to 2017.
1
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
2001
Charity ratings for everyone
Charity Navigator, a US nonprofit, has published free charity ratings since 2001 and says it now rates more than 245,000 charities on measures including financial health and the cost-effectiveness of programs. It charges neither donors nor the charities it rates.
Credit
Charity Navigator
2003
Random trials for aid
Abhijit Banerjee, Esther Duflo and Sendhil Mullainathan founded the Poverty Action Lab, now J-PAL, at MIT in 2003 to test anti-poverty programs with randomized evaluations, in which a lottery decides who gets a program so that the two groups can be compared fairly. J-PAL counts more than 1,300 researchers at over 125 universities in its network, and says Banerjee, Duflo and Michael Kremer shared the 2019 Nobel Prize in economics for this approach.
Credit
Abhijit Banerjee, Esther Duflo and Sendhil Mullainathan (J-PAL)
2013
Scared Straight did harm
"Scared Straight" programs take young offenders or children at risk on organized prison visits to deter them from crime. A Campbell Collaboration review of nine US randomized studies with 946 participants found no evidence the programs work; in the seven studies that reported reoffending, the programs increased the odds of offending.
LessonA well-meant program can make things worse, and only a comparison group shows it.
Credit
Anthony Petrosino, Carolyn Turpin-Petrosino, Meghan E. Hollis-Peel and Julia G. Lavenberg (Campbell Collaboration)
2013
Overhead is a poor guide
The heads of the BBB Wise Giving Alliance, GuideStar and Charity Navigator, which call themselves America's three leading sources of information about charities, wrote jointly to donors that the share of spending on administration and fundraising, called "overhead", "is a poor measure of a charity's performance". They asked donors to weigh transparency, governance, leadership and results instead.
LessonJudging charities by one cost ratio can starve them of the training, planning and evaluation they need to improve, a pattern the letter calls "The Nonprofit Starvation Cycle".
Credit
Art Taylor (BBB Wise Giving Alliance), Jacob Harold (GuideStar), Ken Berger (Charity Navigator)
2015
Microcredit's modest effects
Small loans to poor households and businesses, called microcredit, had long been debated as a way to reduce poverty. In January 2015 the journal AEJ: Applied Economics published six randomized evaluations from six countries on four continents, and its guest editors found "a consistent pattern of modestly positive, but not transformative, effects".
LessonAn idea can be funded at large scale for years before rigorous tests show how much it does.
Credit
Abhijit Banerjee, Dean Karlan and Jonathan Zinman (editors), with the six study teams
2017
Paid only for results
In 2010 the world's first social impact bond was launched at Peterborough prison in England: private investors paid for the One Service, support for prisoners leaving sentences of under 12 months, and the Ministry of Justice, with the Big Lottery Fund, would repay them only if reconvictions fell by at least 7.5%. In 2017 independent evaluators estimated a 9.0% fall in reconviction events across the first two groups of prisoners, enough to trigger payment.
LessonThe estimate for the second group alone was not statistically significant at the usual 95% level, and a change in national probation policy moved the third group to ordinary fee-for-service funding, so the test was cut short.
Credit
Social Finance (which raised the investment); the Ministry of Justice and the Big Lottery Fund (payers); Jake Anders and Richard Dorsett (evaluators)
Watch for
AI as a research critic
GiveWell, which recommends charities based on its own cost-effectiveness research, had AI models critique its research in six grantmaking areas. It reports that about 85% of the critiques were irrelevant or misread its work, that researchers spent about 75 minutes per program sorting them, and that its rough guess for the share of useful critiques had since risen to about 30%. Replying to comments, GiveWell's author wrote that they expect a first AI review pass to be worthwhile "in almost all cases by the end of 2026".
Achieved whenGiveWell or an outside team publishes a repeat test with a fixed method, reporting how many AI critiques changed a cost-effectiveness estimate or a grant.
Claimed by
GiveWell, January 2026
Open
01
Audit AI charity advice
A repeatable test of what AI chatbots tell donors. Put the same set of giving questions, such as "Which charities fight malaria best?", to several chatbots; check every organization they name against official registers; and check every factual claim, such as a cost per life saved or a rating, against its source. Two people check each flagged answer.
The briefCredit · unclaimed02
Track evaluation follow-through
A public check of whether aid agencies did what they promised after evaluations. Many UN agencies publish a "management response" with each evaluation, saying which recommendations they accept and what they will do. An AI model extracts each promised action, then searches the agency's later evaluations, annual reports and board papers for evidence that it happened; a person checks every match.
The briefCredit · unclaimed
Open 01 · the brief
Audit AI charity advice
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A repeatable test of what AI chatbots tell donors. Put the same set of giving questions, such as "Which charities fight malaria best?", to several chatbots; check every organization they name against official registers; and check every factual claim, such as a cost per life saved or a rating, against its source. Two people check each flagged answer.
Why it is still open
The largest study found, Whole Whale's 2026 study of seven AI models answering 255 cause-area questions, counted which organizations the models named and warned that answers "can be wrong", but did not report how many named groups were real, registered and described correctly.
Built with what exists
The chatbots' programming interfaces, run with and without web search; a web agent for chatbots that have none (see the Note "Deleting 80% to start over"); the IRS's free Exempt Organizations Business Master File and the Charity Commission for England and Wales's register download; GiveWell and Charity Navigator pages for checking ratings; Python and a shared spreadsheet.
First step
Put 30 questions to one chatbot, list every organization it names, and check each one against the IRS file by hand.
Done looks like
300 questions put to four chatbots, with and without web search; for every named organization, a record of whether it exists, is registered and is in good standing; error rates by chatbot and by type, with full transcripts published; the test rerun after each major model update.
Open 02 · the brief
Track evaluation follow-through
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public check of whether aid agencies did what they promised after evaluations. Many UN agencies publish a "management response" with each evaluation, saying which recommendations they accept and what they will do. An AI model extracts each promised action, then searches the agency's later evaluations, annual reports and board papers for evidence that it happened; a person checks every match.
Why it is still open
ALNAP's 2026 review, citing a 2023 review for the UN Evaluation Group, found management responses "fairly well institutionalised" across multilateral agencies but follow-up on implementation "weaker". A May 2026 commentary by an external evaluator of UN programs called for moving evaluation budgets "from data collection to follow-through". This research found agencies' own tracking but no independent public check across agencies.
Built with what exists
The evaluation libraries UN agencies publish; pdftotext; an AI model with a long context window, such as Claude or Gemini, to extract actions and search later documents (see the Note "Seven more flagged"); a shared spreadsheet.
First step
Pick five evaluations from 2020 or 2021 at one agency, list the actions it accepted, and look for each in that agency's 2023 to 2025 documents by hand.
Done looks like
300 accepted recommendations from 50 evaluations, each marked done, partly done, not done or no evidence, with the evidence linked; two people agree with the AI's marking on at least 85% of a random 100; each agency sent its results for comment before publication.