BRENNEN

Industries / Government, education and community / Nonprofits

Measuring results

Evaluators, researchers and raters try to find out whether a charity's programs work, what they cost per result, and how charities compare.

6

Milestones achieved, from 2001 to 2017.

1

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 2001

    Charity ratings for everyone

    Charity Navigator, a US nonprofit, has published free charity ratings since 2001 and says it now rates more than 245,000 charities on measures including financial health and the cost-effectiveness of programs. It charges neither donors nor the charities it rates.

    charitynavigator.org

    Credit

    Charity Navigator

  2. 2003

    Random trials for aid

    Abhijit Banerjee, Esther Duflo and Sendhil Mullainathan founded the Poverty Action Lab, now J-PAL, at MIT in 2003 to test anti-poverty programs with randomized evaluations, in which a lottery decides who gets a program so that the two groups can be compared fairly. J-PAL counts more than 1,300 researchers at over 125 universities in its network, and says Banerjee, Duflo and Michael Kremer shared the 2019 Nobel Prize in economics for this approach.

    povertyactionlab.org

    Credit

    Abhijit Banerjee, Esther Duflo and Sendhil Mullainathan (J-PAL)

  3. 2013

    Scared Straight did harm

    "Scared Straight" programs take young offenders or children at risk on organized prison visits to deter them from crime. A Campbell Collaboration review of nine US randomized studies with 946 participants found no evidence the programs work; in the seven studies that reported reoffending, the programs increased the odds of offending.

    LessonA well-meant program can make things worse, and only a comparison group shows it.

    api.crossref.org

    Credit

    Anthony Petrosino, Carolyn Turpin-Petrosino, Meghan E. Hollis-Peel and Julia G. Lavenberg (Campbell Collaboration)

  4. 2013

    Overhead is a poor guide

    The heads of the BBB Wise Giving Alliance, GuideStar and Charity Navigator, which call themselves America's three leading sources of information about charities, wrote jointly to donors that the share of spending on administration and fundraising, called "overhead", "is a poor measure of a charity's performance". They asked donors to weigh transparency, governance, leadership and results instead.

    LessonJudging charities by one cost ratio can starve them of the training, planning and evaluation they need to improve, a pattern the letter calls "The Nonprofit Starvation Cycle".

    charities.org

    Credit

    Art Taylor (BBB Wise Giving Alliance), Jacob Harold (GuideStar), Ken Berger (Charity Navigator)

  5. 2015

    Microcredit's modest effects

    Small loans to poor households and businesses, called microcredit, had long been debated as a way to reduce poverty. In January 2015 the journal AEJ: Applied Economics published six randomized evaluations from six countries on four continents, and its guest editors found "a consistent pattern of modestly positive, but not transformative, effects".

    LessonAn idea can be funded at large scale for years before rigorous tests show how much it does.

    aeaweb.org

    Credit

    Abhijit Banerjee, Dean Karlan and Jonathan Zinman (editors), with the six study teams

  6. 2017

    Paid only for results

    In 2010 the world's first social impact bond was launched at Peterborough prison in England: private investors paid for the One Service, support for prisoners leaving sentences of under 12 months, and the Ministry of Justice, with the Big Lottery Fund, would repay them only if reconvictions fell by at least 7.5%. In 2017 independent evaluators estimated a 9.0% fall in reconviction events across the first two groups of prisoners, enough to trigger payment.

    LessonThe estimate for the second group alone was not statistically significant at the usual 95% level, and a change in national probation policy moved the third group to ordinary fee-for-service funding, so the test was cut short.

    gov.ukassets.publishing.service.gov.uk

    Credit

    Social Finance (which raised the investment); the Ministry of Justice and the Big Lottery Fund (payers); Jake Anders and Richard Dorsett (evaluators)

Watch for

  1. AI as a research critic

    GiveWell, which recommends charities based on its own cost-effectiveness research, had AI models critique its research in six grantmaking areas. It reports that about 85% of the critiques were irrelevant or misread its work, that researchers spent about 75 minutes per program sorting them, and that its rough guess for the share of useful critiques had since risen to about 30%. Replying to comments, GiveWell's author wrote that they expect a first AI review pass to be worthwhile "in almost all cases by the end of 2026".

    Achieved whenGiveWell or an outside team publishes a repeat test with a fixed method, reporting how many AI critiques changed a cost-effectiveness estimate or a grant.

    forum.effectivealtruism.org

    Claimed by

    GiveWell, January 2026

Open

  1. 01

    Audit AI charity advice

    A repeatable test of what AI chatbots tell donors. Put the same set of giving questions, such as "Which charities fight malaria best?", to several chatbots; check every organization they name against official registers; and check every factual claim, such as a cost per life saved or a rating, against its source. Two people check each flagged answer.

    The brief
    Credit · unclaimed
  2. 02

    Track evaluation follow-through

    A public check of whether aid agencies did what they promised after evaluations. Many UN agencies publish a "management response" with each evaluation, saying which recommendations they accept and what they will do. An AI model extracts each promised action, then searches the agency's later evaluations, annual reports and board papers for evidence that it happened; a person checks every match.

    The brief
    Credit · unclaimed

Open 01 · the brief

Audit AI charity advice

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A repeatable test of what AI chatbots tell donors. Put the same set of giving questions, such as "Which charities fight malaria best?", to several chatbots; check every organization they name against official registers; and check every factual claim, such as a cost per life saved or a rating, against its source. Two people check each flagged answer.

Why it is still open

The largest study found, Whole Whale's 2026 study of seven AI models answering 255 cause-area questions, counted which organizations the models named and warned that answers "can be wrong", but did not report how many named groups were real, registered and described correctly.

wholewhale.com

Built with what exists

The chatbots' programming interfaces, run with and without web search; a web agent for chatbots that have none (see the Note "Deleting 80% to start over"); the IRS's free Exempt Organizations Business Master File and the Charity Commission for England and Wales's register download; GiveWell and Charity Navigator pages for checking ratings; Python and a shared spreadsheet.

  1. First step

    Put 30 questions to one chatbot, list every organization it names, and check each one against the IRS file by hand.

  2. Done looks like

    300 questions put to four chatbots, with and without web search; for every named organization, a record of whether it exists, is registered and is in good standing; error rates by chatbot and by type, with full transcripts published; the test rerun after each major model update.

Open 02 · the brief

Track evaluation follow-through

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public check of whether aid agencies did what they promised after evaluations. Many UN agencies publish a "management response" with each evaluation, saying which recommendations they accept and what they will do. An AI model extracts each promised action, then searches the agency's later evaluations, annual reports and board papers for evidence that it happened; a person checks every match.

Why it is still open

ALNAP's 2026 review, citing a 2023 review for the UN Evaluation Group, found management responses "fairly well institutionalised" across multilateral agencies but follow-up on implementation "weaker". A May 2026 commentary by an external evaluator of UN programs called for moving evaluation budgets "from data collection to follow-through". This research found agencies' own tracking but no independent public check across agencies.

alnap.orggppi.net

Built with what exists

The evaluation libraries UN agencies publish; pdftotext; an AI model with a long context window, such as Claude or Gemini, to extract actions and search later documents (see the Note "Seven more flagged"); a shared spreadsheet.

  1. First step

    Pick five evaluations from 2020 or 2021 at one agency, list the actions it accepted, and look for each in that agency's 2023 to 2025 documents by hand.

  2. Done looks like

    300 accepted recommendations from 50 evaluations, each marked done, partly done, not done or no evidence, with the evidence linked; two people agree with the AI's marking on at least 85% of a random 100; each agency sent its results for comment before publication.