BRENNEN

Industries / Finance and business services / Legal and professional services / Consulting

Modeling

Modeling is building a working copy of a business, a project or a market, usually in a spreadsheet, to see what happens to costs, cash or demand when an assumption changes.

7

Milestones achieved, from 1961 to 2026.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1961

    Factories modeled as feedback loops

    In the mid-1950s Jay Forrester of MIT studied a General Electric appliance factory that swung between boom and bust. He built a computer simulation of its supply chain from weekly orders, inventory, production and staff, and showed that management practices, not market forces, caused the cycle. His book Industrial Dynamics (1961) started system dynamics, a way of analyzing an organization by simulating it on a computer.

    news.mit.edu

    Credit

    Jay W. Forrester, MIT Sloan School of Management

  2. 1979

    The electronic spreadsheet arrives

    Dan Bricklin had the idea for an electronic spreadsheet in spring 1978, as a student at Harvard Business School, where he recalls that "10% was always the answer" because complicated calculations took too long by hand. With Bob Frankston he built VisiCalc for the Apple II and announced it in June 1979 at the National Computer Conference. Their site calls it "the first computer spreadsheet program as we know them today."

    bricklin.combricklin.com

    Credit

    Dan Bricklin and Bob Frankston, Software Arts

  3. 1985

    Scenarios instead of a forecast

    Shell's planners began long-term studies of alternative futures in 1967, and in the early 1970s Ted Newland's and Pierre Wack's teams showed managers scenarios in which oil ran short and its price rose sharply. In September 1985 Wack, retired head of Shell's business environment division, explained the method in Harvard Business Review: several stories of how the future could turn out, used to test managers' assumptions, in place of a single forecast.

    LessonForecasts that worked in the stable 1950s and 1960s failed more often, and by more, from the early 1970s, so a plan built on one forecast can break when it matters most.

    hbr.orgpolytechnique-insights.com

    Credit

    Pierre Wack and Edward (Ted) Newland, Royal Dutch/Shell Group planning

  4. 2000

    Most spreadsheets hold errors

    Raymond Panko gathered field audits of real company spreadsheets for the European Spreadsheet Risks Interest Group (EuSpRIG). The most recent audits, which used better methods, found errors in at least 86% of the spreadsheets they checked. In experiments, people checking a spreadsheet cell by cell on their own caught only about half of the errors.

    LessonA model that looks right has usually not been proved right; in 2000 only cell-by-cell inspection had been shown to cut errors.

    arxiv.org

    Credit

    Raymond R. Panko

  5. 2013

    A spreadsheet error, found outside

    Thomas Herndon, Michael Ash and Robert Pollin of the University of Massachusetts Amherst tried to reproduce a 2010 study by Carmen Reinhart and Kenneth Rogoff on public debt and economic growth. Using the authors' own working spreadsheet, they found a coding error that left Australia, Austria, Belgium, Canada and Denmark out of the analysis, along with selectively excluded data and unusual weighting. With all three fixed, average growth for countries with public debt above 90% of GDP was 2.2%, not the published -0.1%.

    LessonErrors in a model are often found only when someone outside gets the working file.

    peri.umass.edu

    Credit

    Thomas Herndon, Michael Ash and Robert Pollin

  6. 2013

    Checks for government models

    After errors in models played a role in the collapse of the UK's InterCity West Coast rail franchise competition in 2012, Sir Nicholas Macpherson, Permanent Secretary to the Treasury, reviewed how government checks its models. His March 2013 report collected information on around 500 business-critical models and took practices from professional firms, noting that in consulting and accountancy firms "it is the norm for a senior partner to sign-off on models prior to external release." In 2022 the National Audit Office wrote that errors in government models "have directly caused significant losses of public money and delays to critical public programmes."

    LessonA model behind a large decision needs one named owner and checks sized to the risk.

    gov.uknao.org.uk

    Credit

    Sir Nicholas Macpherson, HM Treasury

  7. 2026

    Agents build, debugging lags

    SpreadsheetBench, built by Renmin University of China and Tsinghua University from 912 questions people posted on Excel forums, found in 2024 that Copilot in Excel solved about 20% of tasks, against 71.33% for four Excel experts on a 50-question sample. By June 2026 the benchmark's team had run Kingsoft Office's Qingqiu Agent through the maker's own interface at 83.11% on all 912 questions. Their harder SpreadsheetBench 2 (June 2026) sets 321 jobs built from financial reports and company filings; the best model completed 34.89% of them and 12.00% of the jobs that ask it to find and fix errors in a workbook.

    LessonSoftware learned to build formulas faster than it learned to check a whole model.

    arxiv.orgspreadsheetbench.github.ioarxiv.org

    Credit

    Zeyao Ma, Jing Zhang, Jie Tang and colleagues; for version 2, Renmin University of China with Aptura.AI, AfterQuery and Shortcut.AI

Watch for

  1. Agent mode in Excel

    Microsoft says Agent Mode in Excel, which builds and edits workbooks from plain-language requests, completed 57.2% of the 912 SpreadsheetBench tasks, a figure from Microsoft's own run of the benchmark. The SpreadsheetBench leaderboard lists the result as unverified.

    Achieved whenthe SpreadsheetBench team runs Agent Mode itself, as it has for other products, and publishes the score.

    microsoft.comspreadsheetbench.github.io

    Claimed by

    Microsoft, September 2025

  2. AI in a modeling contest

    Anthropic says that when FundamentalLabs built an Excel agent on Claude Opus 4, it passed 5 of 7 levels of the Financial Modeling World Cup, a competition for people who build financial models, and scored 83% on complex Excel tasks.

    Achieved whenthe competition's organizers or another outside party publish an AI entrant's results from a live round run under the same rules and time limits as the human entrants.

    anthropic.com

    Claimed by

    Anthropic, citing FundamentalLabs, July 2025

Open

  1. 01

    Planted errors, people versus AI

    A public test of model checking. Take 30 models of the kinds consultants build, such as cost models, business cases and cash forecasts, plant known errors of each common type in copies, and give the same copies to AI reviewers and to experienced human model reviewers. Count what each catches, by error type, and how often each flags a cell that is correct.

    The brief
    Credit · unclaimed
  2. 02

    An audit of public models

    Collect spreadsheet models that governments publish alongside policy papers and business cases, have an AI agent explain and test every formula, have a person confirm each suspected error, and publish an error count per model in the style of the field audits Panko collected in 2000.

    The brief
    Credit · unclaimed

Open 01 · the brief

Planted errors, people versus AI

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public test of model checking. Take 30 models of the kinds consultants build, such as cost models, business cases and cash forecasts, plant known errors of each common type in copies, and give the same copies to AI reviewers and to experienced human model reviewers. Count what each catches, by error type, and how often each flags a cell that is correct.

Why it is still open

The tests so far are small and have no human reviewers on the same files. FLARE, presented at EuSpRIG in 2025, seeded errors in two small spreadsheets. An ISPOR Europe 2026 poster planted 72 errors in one health-economics model and tested one AI model, which found 58 of 72 by majority of three runs and missed errors where a valid formula returned a believable number. SpreadsheetBench 2 scores agents at fixing errors but has no human baseline.

ispor.orgarxiv.org

Built with what exists

the open-source openpyxl library to plant errors and list every formula; LibreOffice in headless mode to recalculate; AI reviewers such as Claude for Excel and Microsoft's Agent Mode in Excel, plus a scripted agent through an API; Panko's error types (mechanical, logic and omission) as the starting list; each AI reviewer run five times on every file, counting errors caught on every run (see the Note "All twenty times").

  1. First step

    Plant 20 errors in one public cost model and time two human reviewers and one AI tool on it.

  2. Done looks like

    30 models, at least 300 planted errors across at least six types, at least 10 human reviewers and 3 AI tools; catch rate and false-flag rate per error type published with 95% confidence intervals, and every file released.

Open 02 · the brief

An audit of public models

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

Collect spreadsheet models that governments publish alongside policy papers and business cases, have an AI agent explain and test every formula, have a person confirm each suspected error, and publish an error count per model in the style of the field audits Panko collected in 2000.

Why it is still open

The field audits Panko summarized date from 1987 to 2000 and cover private spreadsheets. The UK National Audit Office wrote in 2022 that errors in government models have caused significant losses of public money and called for greater transparency around models, and searches in this session found no published cell-by-cell audit of models that governments release.

nao.org.ukarxiv.org

Built with what exists

Python and openpyxl to extract every formula; Claude for Excel or Agent Mode in Excel to explain formulas and suggest checks; an AI coding agent to write consistency tests; a shared log that a second person audits.

  1. First step

    Find 10 spreadsheet models that one government has published with policy papers this year and count their formula cells.

  2. Done looks like

    50 public models checked, every flagged error confirmed by two people and sent to the publishing department for a reply before release, and a published tally of models with at least one confirmed error and of errors per 1,000 formula cells.