BRENNEN

Industries / Finance and business services / Advertising and marketing / Market research

Surveys

Surveys ask many people the same written questions, in the same order, so that their answers can be counted.

5

Milestones achieved, from 1932 to 2020.

3

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1932

    Agree-or-disagree scales

    Rensis Likert of New York University had people mark each statement on a five-step range, from "strongly approve" through "undecided" to "strongly disapprove," and scored each person by adding or averaging numbers given to the steps. He reported that this simple scoring gave "essentially the same reliability" as a more elaborate statistical method, and matched the reliability of the Thurstone-Droba war-attitudes scale, built by Louis Thurstone's method, with "practically one-half as many items."

    archive.org

    Credit

    Rensis Likert, Archives of Psychology No. 140, June 1932

  2. 1941

    Wording changes the answer

    Donald Rugg split a sample at random and asked one half whether the United States should "allow" public speeches against democracy and the other half whether it should "forbid" them. By the figures CBS News reports, 75% would "not allow" such speeches but only 54% would "forbid" them. Researchers later tested the pair in West Germany and the Netherlands and found a gap of the same kind in New Zealand, but a 1989 US General Social Survey test about a speech by an "admitted communist" found none: 34% would forbid it and 35% would not allow it.

    LessonTwo questions that mean the same thing in logic can draw different answers, and the gap has to be measured each time, not assumed.

    cbsnews.com

    Credit

    Donald Rugg, Public Opinion Quarterly, 1941

  3. 1994

    Surveys move to the web

    In January 1994 James Pitkow and Margaret Recker of the Georgia Institute of Technology posted a questionnaire on the World Wide Web for a month, using HTML forms, which let people answer by clicking. It drew 4,777 respondents, while an email survey about Web technologies posted to the comp.infosystems.www newsgroup in late January 1994 drew 55 replies. Georgia Tech's team later called theirs "the first publicly accessible Web-based survey."

    LessonThe authors warned that their respondents were "a particular class of WWW users" and "cannot yet be generalized," the same limit that self-selected online samples still have.

    sites.cc.gatech.edusites.cc.gatech.edu

    Credit

    James Pitkow and Margaret Recker, GVU Center, Georgia Tech

  4. 2018

    Phone response rates reach 6%

    Pew Research Center reported that typical response rates to its telephone surveys fell to 7% in 2017 and 6% in 2018, after holding near 9% for several years. Its own studies in 1997, 2003, 2012 and 2016 found little link between response rate and accuracy, but falling response raised costs, and Pew moved most of its US polling to its online American Trends Panel.

    LessonA low response rate does not by itself make a survey wrong, but in Pew's words it "does signal that the risk of error is higher.".

    pewresearch.org

    Credit

    Pew Research Center

  5. 2020

    Phone surveys replace visits

    When COVID-19 stopped face-to-face interviewing, the World Bank's Living Standards Measurement Study (LSMS) program supported 61 monthly phone survey rounds in 7 countries between April 2020 and June 2021, more than 119,000 completed interviews, and published the data from each round. The surveys called households from earlier face-to-face surveys that had recorded phone numbers, so what was already known about each household could be used to check for the people a phone survey misses.

    LessonWhere many people have no phone, an earlier in-person survey that recorded phone numbers is what makes a fast phone survey possible and checkable.

    pmc.ncbi.nlm.nih.gov

    Credit

    the World Bank LSMS team; account by Sydney Gourlay, Talip Kilic, Antonio Martuscelli, Philip Wollburg and Alberto Zezza in Food Policy

Watch for

  1. Chatbots that ask follow-ups

    Researchers at NORC at the University of Chicago randomly assigned 1,800 web survey participants to text chatbots that asked them to say more and coded their written answers as they went. They report "more detailed and informative" answers, live coding that worked "moderately well," and "a slight cost to respondent experience." A separate team led by Alexander Wuttke compared AI and human interviewers with university students and found the AI produced "quality data comparable to traditional methods."

    Achieved whena team outside both groups repeats the test on a general-population sample, or a large production survey publishes its results against a randomized arm that gets the standard questions.

    ojs.ub.uni-konstanz.deaclanthology.org

    Claimed by

    Soubhik Barari and colleagues, NORC, Survey Research Methods, 2026 (first posted April 2025); Wuttke and colleagues, May 2025

  2. AI voice phone interviews

    A team working with the SSRS Opinion Panel built an AI system that conducts phone surveys while keeping exact question wording and randomizing question and answer order, and ran two pilot surveys with panel members. They measured completion rates, break-off rates and satisfaction, and say shorter questionnaires and more responsive AI interviewers "may contribute to improvements across all three."

    Achieved whena randomized comparison with human interviewers on the same questionnaire is published, preferably by someone other than the system's builders.

    arxiv.org

    Claimed by

    Danny D. Leybzon and six co-authors, July 2025

  3. AI checks draft questions

    Patrick Sturgis, Caroline Roberts and Thomas Robinson tested AI models as stand-ins for cognitive pretesting, in which interviewers ask a few people to think aloud while they answer draft questions. On 20 questions with problems built in on purpose, the best set-up found 75% of the known problems "with a small false positive rate," and testing a 20-item questionnaire took under an hour and cost "a few dollars."

    Achieved whenthe method finds the problems that real cognitive interviews found, on questions its authors did not write.

    iser.essex.ac.uk

    Claimed by

    Sturgis, Roberts and Robinson, Survey Futures working paper 15, University of Essex, April 2026

Open

  1. 01

    Score AI pretests against interviews

    A test that runs AI question checkers on survey questions that have already been through real cognitive interviews, and scores how many of the documented problems each checker finds and how many it invents.

    The brief
    Credit · unclaimed
  2. 02

    Randomize AI and human interviewers

    A phone survey in which people who have agreed to take research calls are split at random, half interviewed by an AI voice interviewer and half by trained human interviewers, with the same questionnaire.

    The brief
    Credit · unclaimed

Open 01 · the brief

Score AI pretests against interviews

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A test that runs AI question checkers on survey questions that have already been through real cognitive interviews, and scores how many of the documented problems each checker finds and how many it invents.

Why it is still open

The best published test, from the University of Essex in April 2026, scored AI models on 20 questions with problems planted by the authors, plus 7 control questions from the European Social Survey. Q-Bank, run by the US Centers for Disease Control and Prevention, holds reports from real question-evaluation studies, and searches found no test of AI checkers against them.

iser.essex.ac.ukwwwn.cdc.gov

Built with what exists

Q-Bank's question evaluation reports; the prompt designs described in the Essex paper; open-weight and commercial AI models through their APIs; Python for scoring.

  1. First step

    Download 20 Q-Bank reports on different topics and have two people list, question by question, the problems each report documents.

  2. Done looks like

    At least 200 questions from at least 20 Q-Bank reports, with the documented problems listed by two people working separately before any model is run, and each checker's detection and false-alarm rates published with 95% confidence intervals, prompts and code.

Open 02 · the brief

Randomize AI and human interviewers

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A phone survey in which people who have agreed to take research calls are split at random, half interviewed by an AI voice interviewer and half by trained human interviewers, with the same questionnaire.

Why it is still open

The SSRS pilots compared two runs of the AI system and followed up with a separate human-administered survey about the experience; they had no randomized human-interviewer arm. Wuttke's comparison used university students in conversational interviews, not a phone poll.

arxiv.orgaclanthology.org

Built with what exists

A speech-to-text model with turn detection, such as ElevenLabs Scribe v3 Live (see the Note "Fewer word errors"); a current AI model as the interviewer; a text-to-speech voice; a phone service such as Twilio; a survey panel whose members consent to automated research calls; a human calling team hired from a field vendor.

  1. First step

    Pre-register the design on the Open Science Framework (OSF) and get quotes from two panel providers for 1,000 completed phone interviews with consent to automated calls.

  2. Done looks like

    At least 500 completed interviews in each arm, with response, break-off and item nonresponse rates, and differences on at least 20 questions whose population values are known, published with a pre-registered analysis and code.