Industries / Finance and business services / Advertising and marketing / Market research
Panels and sampling
Sampling is choosing whom to ask, and a panel is a standing pool of people who have agreed to answer surveys again and again.
7
Milestones achieved, from 1934 to 2023.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1934
Random beats hand-picked
The Italian statisticians C. Gini and L. Galvani had kept a sample of Italy's 1921 census by choosing 29 of its 214 districts so that the sample's averages matched the country on seven measures. Jerzy Neyman pointed out that, by the authors' own comparison, the sample agreed on those seven but agreement on other characteristics "is often poor," while random stratified sampling gives an estimate whose accuracy can be stated "whatever the properties of the population."
LessonA sample matched on a few known totals can still be wrong on everything else.
Credit
Jerzy Neyman, Journal of the Royal Statistical Society, vol. 97, 1934
1948
Quotas miss Truman
The Crossley, Gallup and Roper polls all put Thomas Dewey ahead of Harry Truman, by 50% to 45%, 50% to 44% and 53% to 38%. Truman won with 50% to Dewey's 45%. All three used quota sampling, in which interviewers must fill set numbers by area, sex, age, race and economic status but may choose whom to approach, and that freedom is "now generally accepted" to have favored Dewey.
LessonFilling quotas does not remove the bias of letting interviewers pick the respondents.
Credit
the Crossley, Gallup and Roper organizations
1978
Random dialing made efficient
Joseph Waksberg published a way to pick US household telephone numbers at random without wasting most calls on numbers that reach no home. Numbers are grouped in banks of 100, one random number in a bank is called, and the bank is kept for more calls only if that number reaches a household. The method, named for Warren Mitofsky as well, is among the most cited in survey sampling: Crossref counts 936 citations of the paper.
Credit
Joseph Waksberg and Warren Mitofsky; "Sampling Methods for Random Digit Dialing," Journal of the American Statistical Association, March 1978
2007
A web panel for everyone
For the LISS panel in the Netherlands, a simple random sample of 10,150 addresses was drawn with Statistics Netherlands, and interviewers recruited households by phone and in person from May to December 2007. Households without internet access were lent a computer and a broadband connection, so that people without the internet could belong to an online panel.
LessonA panel can answer online and still be chosen at random, if the panel supplies the equipment.
Credit
CentERdata, Tilburg; fieldwork by TNS NIPO; account by Annette Scherpenzeel
2016
UK polls miss on samples
Polls on the eve of the 2015 UK general election pointed to a dead heat, and the Conservatives beat Labour by seven points. An inquiry for the British Polling Council and the Market Research Society, chaired by Patrick Sturgis, found that the primary cause was unrepresentative samples. It made 12 recommendations, including pre-registering vote-intention polls before fieldwork and publishing significance tests for changes between polls.
Credit
Patrick Sturgis and the inquiry panel
2021
Nonresponse defeats weighting
A task force of the American Association for Public Opinion Research (AAPOR) found that the 2020 US election polls had their largest error in 40 years for the national popular vote, overstating Joe Biden's margin over Donald Trump by 3.9 points in national polls and 4.3 points in state presidential polls. It found that "at least some" of the error came from people who did not respond, and that adjustments for demographics, party and 2016 vote, which had worked before, "did not render accurate vote estimates." By AAPOR's 2025 review, average error across 611 polls fell to 3.3 points in 2024, from 5.3 in 2020, though polls underestimated Republicans for a third presidential cycle in a row.
LessonWhen the people who refuse a survey differ in the very thing being measured, weighting on known traits cannot fully correct it.
Credit
AAPOR Task Force on 2020 Pre-Election Polling
2023
Opt-in samples, double the error
Pew Research Center asked the same questions of three panels recruited at random by mail and three opt-in samples, whose members come from sources such as social media ads and reward websites, and compared the answers with 28 benchmarks from government data. Average error was 2.6 percentage points for the random panels and 5.8 for the opt-in samples; for adults aged 18 to 29 it was 11.2 points in the opt-in samples against 3.6.
LessonThe same weighting scheme was applied to all six samples, and it did not close the gap.
Credit
Andrew Mercer and Arnold Lau, Pew Research Center
Watch for
People simulated from interviews
Joon Sung Park and ten co-authors interviewed 1,052 US adults for two hours each with an AI voice interviewer and built an AI simulation of each person. On held-out General Social Survey questions, simulations built from interviews and surveys together matched people's answers 72.8% of the time, against 79.5% for people repeating their own answers two weeks later and 60.4% for simulations built from demographics alone. Park is chief executive of Simile, which sells such simulations and raised a US$200 million round in July 2026.
Achieved whenan outside team scores the simulations on survey waves fielded after the simulations are made; Pew's own 2026 test of AI "digital twins," built differently, missed human results by 12 points on average.
Claimed by
Park and co-authors, latest revision November 2026
Synthetic panels for hire
Qualtrics says its Edge Audiences model, an AI model fine-tuned on "millions of human respondents," reproduced a published study of attitudes to Google Search with 584 synthetic answers whose means differed from the human ones by 0.07 standard deviations on average, against 0.87 for GPT and 0.88 for Gemini, which it calls "12x more accurate." In March 2026 it began selling synthetic consumer panels for US audiences.
Achieved whenan outside party tests the panels on many studies of its own choosing, whose human results are not yet public when the synthetic answers are made.
Claimed by
Qualtrics, October 2025 and March 2026
Open
01
Test synthetic boosts on holdouts
An independent, pre-registered test of "boosting," in which a model adds synthetic respondents to a small subgroup of a survey. The test hides most real members of each subgroup, asks each method to fill the gap from the rest of the data, and compares the boosted estimates with the hidden respondents' real answers.
The briefCredit · unclaimed02
Benchmark opt-in panels further afield
A published accuracy test of opt-in online panels in large countries where none has been published, such as India, Indonesia and Nigeria, comparing panel answers with benchmarks from official household surveys.
The briefCredit · unclaimed
Open 01 · the brief
Test synthetic boosts on holdouts
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
An independent, pre-registered test of "boosting," in which a model adds synthetic respondents to a small subgroup of a survey. The test hides most real members of each subgroup, asks each method to fill the gap from the rest of the data, and compares the boosted estimates with the hidden respondents' real answers.
Why it is still open
The checks published so far are run by firms that sell or use the method. Big Village, a US research firm, describes an "internal proof of concept" in which it compared synthetic data for 10 niche audiences with a holdout sample of real respondents.
Built with what exists
Large public survey files such as the European Social Survey and Pew Research Center datasets; open synthetic-data tools such as the synthpop package for R and the Synthetic Data Vault for Python as baselines; commercial boosting services under trial licenses; Python for scoring.
First step
Pick 20 small subgroups in the latest European Social Survey round and write the pre-registration, including which answers will be hidden.
Done looks like
For at least 20 subgroups of 50 to 150 people, the average error of boosted and unboosted estimates against the hidden answers, and how often each method's stated margin of error contains the real value, published with the code.
Open 02 · the brief
Benchmark opt-in panels further afield
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A published accuracy test of opt-in online panels in large countries where none has been published, such as India, Indonesia and Nigeria, comparing panel answers with benchmarks from official household surveys.
Why it is still open
Pew's 2023 test covered the United States. A study in PS: Political Science & Politics, published online in January 2023, ran nine online surveys in six Latin American countries and found an average error of 8.72 points on benchmark questions; its authors wrote that such samples had been examined "in the United States and Europe" but that little was known elsewhere. Searches found no comparable test for India, Indonesia or Nigeria.
Built with what exists
Benchmarks from national statistics offices and the Demographic and Health Surveys (DHS) program; samples bought at standard rates from at least three panel providers per country; the mean-absolute-error method used by Pew and the Latin American study; Python or R.
First step
List 20 questions asked in the latest official surveys of all three countries and request quotes from panel providers.
Done looks like
For each of three countries and three providers, at least 1,000 completed surveys and 15 benchmark questions, with average error overall and for rural, older and less-educated respondents, and the data published.