BRENNEN

Industries / Government, education and community / Nonprofits / Think tanks

Research

Think tank researchers collect evidence, run surveys and build models to estimate what a policy would do and who it would affect.

6

Milestones achieved, from 1907 to 2016.

1

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1907

    Surveying a steel city

    In May 1907, weeks after Margaret Olivia Sage set it up with $10 million, the Russell Sage Foundation funded the Pittsburgh Survey. The foundation calls it the first systematic effort to survey the employment and living conditions of the working class in a large US city, and says its findings led to new rules on working conditions and employers' liability and helped end twelve-hour days and seven-day weeks for steel workers.

    russellsage.org

    Credit

    Margaret Olivia Sage; the Russell Sage Foundation

  2. 1965

    Same interviews, opposite readings

    In Vietnam, RAND interviewed Viet Cong prisoners and defectors about their motivation and morale, more than 1,000 interviews by January 1967. The first phase, led by Joseph Zasloff and John Donnell in 1964, described a resilient enemy, and their report never reached Defense Secretary Robert McNamara. Leon Goure, who led the project from 1965 to 1967, told McNamara that bombing had lowered enemy morale; several colleagues were, in the words of RAND's own history, "convinced that Goure was interpreting selectively from the interviews".

    LessonThe same interviews supported opposite conclusions, and the reading that matched what officials hoped to hear traveled further.

    rand.org

    Credit

    the RAND Corporation; Mai Elliott, whose history of the project RAND published in 2010

  3. 1973

    Simulating every household

    The Urban Institute's Transfer Income Model (TRIM) first ran in 1973. It is a microsimulation model: a program that applies the rules of tax and benefit programs to each household in a large survey, here the Census Bureau's Current Population Survey, and adds up the results. It descends from a 1969 model and has been in continuous use since, with its main funding from the US Department of Health and Human Services, for work on tax reform in the 1980s and welfare and health reform in the 1990s.

    urban.org

    Credit

    the Urban Institute; the HHS Office of the Assistant Secretary for Planning and Evaluation

  4. 1999

    Asking Africans directly

    Michael Bratton, Robert Mattes and E. Gyimah-Boadi merged three survey projects into Afrobarometer, with Michigan State University, the Institute for Democracy in South Africa and the Center for Democratic Development in Ghana as core partners. Its first round covered 12 countries and its tenth covers 38. A national partner in each country interviews a random sample of 1,200 to 2,400 adults face to face, in the language each person chooses, which Afrobarometer says gives a margin of error of plus or minus 2 to 3 percentage points.

    afrobarometer.orgafrobarometer.org

    Credit

    Michael Bratton, Robert Mattes and E. Gyimah-Boadi; Afrobarometer's national partners

  5. 2009

    Counting trade barriers

    In June 2009, amid fears that the financial crisis would bring back the trade barriers of the 1930s, the Global Trade Alert began recording government measures that affect trade. Each entry links to the official source so readers can check it, and the project counts tens of thousands of measures since. In January 2024 Simon Evenett, who founded the Swiss St. Gallen Endowment that runs it, published with three co-authors an IMF working paper introducing the New Industrial Policy Observatory, a dataset of the industrial policy measures governments took in 2023.

    globaltradealert.orgdoi.org

    Credit

    Simon Evenett; the St. Gallen Endowment for Prosperity Through Trade

  6. 2016

    Tax models made public

    The American Enterprise Institute said in 2016 that most tax models used to estimate what bills cost were proprietary, so outsiders could not check their assumptions. In April 2016 its Open Source Policy Center launched TaxBrain, a web app that runs open-source tax models written in Python, Route Fifty reported. Its core model, Tax-Calculator, has been public on GitHub since July 2014, and PolicyEngine, a nonprofit, has published open models of UK and US taxes and benefits since 2020 and 2021.

    route-fifty.comgithub.comgithub.com

    Credit

    Matt Jensen and the Open Source Policy Center at the American Enterprise Institute; Tax-Calculator's contributors; PolicyEngine

Watch for

  1. Reports drafted by AI models

    RAND published a report on forecasts of artificial general intelligence whose "primary drafting" was done by three AI models, GPT 5.1, Gemini 3 Pro and Claude 4.5 Opus, with researchers supplying direction, peer review, fact-checking and revision. The authors say this let a small team produce a broader synthesis faster. They also list what human review caught in the AI drafts: wrong or invented citations, outdated facts and overstated confidence, and they say every citation was checked against its source.

    Achieved whenAn outside team checks the claims and citations of a set of AI-drafted think tank reports against their sources and finds error rates no higher than in comparable reports written by people.

    rand.org

    Claimed by

    Gopal P. Sarma, Sunny D. Bhatt, Michael Jacob and Rachel Steratore, RAND, March 2026

Open

  1. 01

    Rerun the reports' numbers

    A public tally of whether the headline numbers in think tank reports can be reproduced from the data, code or spreadsheets their authors published. Sample evenly across the political spectrum and across at least three countries.

    The brief
    Credit · unclaimed
  2. 02

    Score the predictions in reports

    A dataset of dated, checkable predictions made in think tank reports ("X will happen by 2025"), each later marked right, wrong or too vague to score, with a calibration score for organizations that state probabilities.

    The brief
    Credit · unclaimed

Open 01 · the brief

Rerun the reports' numbers

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public tally of whether the headline numbers in think tank reports can be reproduced from the data, code or spreadsheets their authors published. Sample evenly across the political spectrum and across at least three countries.

Why it is still open

Large reproduction efforts so far cover journal articles. Brodeur and colleagues reproduced 110 articles from leading economics and political science journals that require data and code sharing, and found more than 85% of the published claims computationally reproducible (Nature, 2026). The World Bank has checked the data and code behind its own Policy Research Working Papers before release since a 2023 pilot, required it from 2025, and published its 500th verified package in June 2026. This research found no comparable outside check of think tank reports, which are rarely bound by such sharing rules.

iffs.seblogs.worldbank.org

Built with what exists

The reports' posted data and spreadsheets; Python or R; an AI coding agent such as Claude Code to turn methods sections into code, with every step logged for review; Zenodo to archive the results and code.

  1. First step

    List the 100 most recent reports from 20 think tanks, chosen to balance political leanings and countries, and record which of them post data and code.

  2. Done looks like

    60 reports from 20 think tanks attempted, each scored as reproduced, reproduced with differences, or not reproducible because materials are missing; two reviewers agree on every score; each think tank shown its result and given a chance to reply before publication.

Open 02 · the brief

Score the predictions in reports

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A dataset of dated, checkable predictions made in think tank reports ("X will happen by 2025"), each later marked right, wrong or too vague to score, with a calibration score for organizations that state probabilities.

Why it is still open

Forecasting research scores people who answer set questions in tournaments. The scorecards of published forecasts found here are self-graded: Worldview, the geopolitical forecasting service published by RANE, calls its yearly review of its own forecasts "a candid internal assessment". The Forecasting Research Institute's LEAP panel collects new expert forecasts on AI every month to score later; it does not score claims already printed in reports. The closest outside check found is a 2011 Hamilton College class, led by P. Gary Wyckoff, that scored 472 predictions made by 26 newspaper columnists and Sunday television guests from September 2007 to December 2008. This research found no outside scorecard of the predictions in think tank publications.

newswise.comworldview.ranenetwork.comleap.forecastingresearch.orgfp21.substack.com

Built with what exists

An AI model to flag candidate predictions in archived reports, with people confirming each one (see the Note "A thousand questions, one pass"); public statistics such as World Bank indicators to settle them; the Brier score, a standard measure of how accurate probability forecasts are.

  1. First step

    Take 50 reports published in 2018 by five think tanks, mark every dated prediction, and test whether two reviewers agree on what counts as one.

  2. Done looks like

    At least 500 predictions due by 2025, from 20 think tanks balanced across the political spectrum; two reviewers agree on at least 90% of the verdicts; results and scoring rules published by organization.