Industries / Finance and business services / Advertising and marketing / Market research
Surveys
Surveys ask many people the same written questions, in the same order, so that their answers can be counted.
5
Milestones achieved, from 1932 to 2020.
3
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1932
Agree-or-disagree scales
Rensis Likert of New York University had people mark each statement on a five-step range, from "strongly approve" through "undecided" to "strongly disapprove," and scored each person by adding or averaging numbers given to the steps. He reported that this simple scoring gave "essentially the same reliability" as a more elaborate statistical method, and matched the reliability of the Thurstone-Droba war-attitudes scale, built by Louis Thurstone's method, with "practically one-half as many items."
Credit
Rensis Likert, Archives of Psychology No. 140, June 1932
1941
Wording changes the answer
Donald Rugg split a sample at random and asked one half whether the United States should "allow" public speeches against democracy and the other half whether it should "forbid" them. By the figures CBS News reports, 75% would "not allow" such speeches but only 54% would "forbid" them. Researchers later tested the pair in West Germany and the Netherlands and found a gap of the same kind in New Zealand, but a 1989 US General Social Survey test about a speech by an "admitted communist" found none: 34% would forbid it and 35% would not allow it.
LessonTwo questions that mean the same thing in logic can draw different answers, and the gap has to be measured each time, not assumed.
Credit
Donald Rugg, Public Opinion Quarterly, 1941
1994
Surveys move to the web
In January 1994 James Pitkow and Margaret Recker of the Georgia Institute of Technology posted a questionnaire on the World Wide Web for a month, using HTML forms, which let people answer by clicking. It drew 4,777 respondents, while an email survey about Web technologies posted to the comp.infosystems.www newsgroup in late January 1994 drew 55 replies. Georgia Tech's team later called theirs "the first publicly accessible Web-based survey."
LessonThe authors warned that their respondents were "a particular class of WWW users" and "cannot yet be generalized," the same limit that self-selected online samples still have.
Credit
James Pitkow and Margaret Recker, GVU Center, Georgia Tech
2018
Phone response rates reach 6%
Pew Research Center reported that typical response rates to its telephone surveys fell to 7% in 2017 and 6% in 2018, after holding near 9% for several years. Its own studies in 1997, 2003, 2012 and 2016 found little link between response rate and accuracy, but falling response raised costs, and Pew moved most of its US polling to its online American Trends Panel.
LessonA low response rate does not by itself make a survey wrong, but in Pew's words it "does signal that the risk of error is higher.".
Credit
Pew Research Center
2020
Phone surveys replace visits
When COVID-19 stopped face-to-face interviewing, the World Bank's Living Standards Measurement Study (LSMS) program supported 61 monthly phone survey rounds in 7 countries between April 2020 and June 2021, more than 119,000 completed interviews, and published the data from each round. The surveys called households from earlier face-to-face surveys that had recorded phone numbers, so what was already known about each household could be used to check for the people a phone survey misses.
LessonWhere many people have no phone, an earlier in-person survey that recorded phone numbers is what makes a fast phone survey possible and checkable.
Credit
the World Bank LSMS team; account by Sydney Gourlay, Talip Kilic, Antonio Martuscelli, Philip Wollburg and Alberto Zezza in Food Policy
Watch for
Chatbots that ask follow-ups
Researchers at NORC at the University of Chicago randomly assigned 1,800 web survey participants to text chatbots that asked them to say more and coded their written answers as they went. They report "more detailed and informative" answers, live coding that worked "moderately well," and "a slight cost to respondent experience." A separate team led by Alexander Wuttke compared AI and human interviewers with university students and found the AI produced "quality data comparable to traditional methods."
Achieved whena team outside both groups repeats the test on a general-population sample, or a large production survey publishes its results against a randomized arm that gets the standard questions.
Claimed by
Soubhik Barari and colleagues, NORC, Survey Research Methods, 2026 (first posted April 2025); Wuttke and colleagues, May 2025
AI voice phone interviews
A team working with the SSRS Opinion Panel built an AI system that conducts phone surveys while keeping exact question wording and randomizing question and answer order, and ran two pilot surveys with panel members. They measured completion rates, break-off rates and satisfaction, and say shorter questionnaires and more responsive AI interviewers "may contribute to improvements across all three."
Achieved whena randomized comparison with human interviewers on the same questionnaire is published, preferably by someone other than the system's builders.
Claimed by
Danny D. Leybzon and six co-authors, July 2025
AI checks draft questions
Patrick Sturgis, Caroline Roberts and Thomas Robinson tested AI models as stand-ins for cognitive pretesting, in which interviewers ask a few people to think aloud while they answer draft questions. On 20 questions with problems built in on purpose, the best set-up found 75% of the known problems "with a small false positive rate," and testing a 20-item questionnaire took under an hour and cost "a few dollars."
Achieved whenthe method finds the problems that real cognitive interviews found, on questions its authors did not write.
Claimed by
Sturgis, Roberts and Robinson, Survey Futures working paper 15, University of Essex, April 2026
Open
01
Score AI pretests against interviews
A test that runs AI question checkers on survey questions that have already been through real cognitive interviews, and scores how many of the documented problems each checker finds and how many it invents.
The briefCredit · unclaimed02
Randomize AI and human interviewers
A phone survey in which people who have agreed to take research calls are split at random, half interviewed by an AI voice interviewer and half by trained human interviewers, with the same questionnaire.
The briefCredit · unclaimed
Open 01 · the brief
Score AI pretests against interviews
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A test that runs AI question checkers on survey questions that have already been through real cognitive interviews, and scores how many of the documented problems each checker finds and how many it invents.
Why it is still open
The best published test, from the University of Essex in April 2026, scored AI models on 20 questions with problems planted by the authors, plus 7 control questions from the European Social Survey. Q-Bank, run by the US Centers for Disease Control and Prevention, holds reports from real question-evaluation studies, and searches found no test of AI checkers against them.
Built with what exists
Q-Bank's question evaluation reports; the prompt designs described in the Essex paper; open-weight and commercial AI models through their APIs; Python for scoring.
First step
Download 20 Q-Bank reports on different topics and have two people list, question by question, the problems each report documents.
Done looks like
At least 200 questions from at least 20 Q-Bank reports, with the documented problems listed by two people working separately before any model is run, and each checker's detection and false-alarm rates published with 95% confidence intervals, prompts and code.
Open 02 · the brief
Randomize AI and human interviewers
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A phone survey in which people who have agreed to take research calls are split at random, half interviewed by an AI voice interviewer and half by trained human interviewers, with the same questionnaire.
Why it is still open
The SSRS pilots compared two runs of the AI system and followed up with a separate human-administered survey about the experience; they had no randomized human-interviewer arm. Wuttke's comparison used university students in conversational interviews, not a phone poll.
Built with what exists
A speech-to-text model with turn detection, such as ElevenLabs Scribe v3 Live (see the Note "Fewer word errors"); a current AI model as the interviewer; a text-to-speech voice; a phone service such as Twilio; a survey panel whose members consent to automated research calls; a human calling team hired from a field vendor.
First step
Pre-register the design on the Open Science Framework (OSF) and get quotes from two panel providers for 1,000 completed phone interviews with consent to automated calls.
Done looks like
At least 500 completed interviews in each arm, with response, break-off and item nonresponse rates, and differences on at least 20 questions whose population values are known, published with a pre-registered analysis and code.