Industries / Finance and business services / Advertising and marketing / Market research
Data analysis
Data analysis turns raw answers into findings: coding written answers into categories, weighting samples, and modeling what drives choices.
7
Milestones achieved, from 1971 to 2026.
1
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1971
Trade-offs put in numbers
Paul Green of the Wharton School and Vithala Rao brought conjoint measurement into marketing research. People rank or choose among whole product descriptions, and the analysis works out how much each separate feature counted in their choices.
Credit
Paul E. Green and Vithala R. Rao, "Conjoint Measurement for Quantifying Judgmental Data," Journal of Marketing Research, August 1971
1989
A hotel designed by survey
Marriott used conjoint analysis to choose target customers, positioning, room layout and services for a new chain, Courtyard by Marriott, which it test marketed and then opened nationally. The published account says the study also "changed Marriott's approach to new product development."
Credit
Jerry Wind, Paul E. Green, Douglas Shifflet and Marsha Scarbrough, Interfaces, 1989
1994
A national satisfaction index
Researchers at the University of Michigan, with the American Society for Quality and CFI Group, started the American Customer Satisfaction Index (ACSI) in 1994 and first published it that October, with updates each quarter. It was adapted from the Swedish Customer Satisfaction Barometer of 1989; Claes Fornell developed the model for both.
Credit
Claes Fornell
2007
One number fails a retest
In the Harvard Business Review in December 2003, Frederick Reichheld of Bain & Company promoted the Net Promoter score as "the one number you need to grow." Timothy L. Keiningham and three co-authors repeated the analysis with Norwegian data on 21 firms and more than 15,500 interviews, and found the research "fails to replicate his assertions regarding the 'clear superiority'" of the measure over others.
LessonA measure tied to growth by its author needs testing by others, on other data, before it is trusted.
Credit
Keiningham, Cooil, Andreassen and Aksoy, Journal of Marketing, July 2007
2017
Seat estimates from polls
YouGov combined about 50,000 interviews a week with multilevel regression and post-stratification (MRP), a model that estimates opinion in small areas, here each parliamentary constituency, from a large national sample. Its final estimate before the 2017 UK election gave the Conservatives 302 seats and Labour 269, a hung parliament. The result was a hung parliament, with 317 Conservative and 262 Labour seats.
Credit
YouGov
2024
AI codes written answers
Jonathan Mellon and five co-authors compared six AI models with a second human coder at sorting British Election Study answers about "the most important issue" into 50 categories. With no training examples, the best model, Claude-1.3, was right 93.9% of the time against the human's 94.7%. A 2025 study of German answers about why people take part in surveys, by Leah von der Heyde and three co-authors, found that "only a fine-tuned LLM achieves satisfactory levels of predictive performance."
LessonHow well AI codes answers depends on the language, the topic and the model, so one good result does not carry over.
Credit
Mellon et al., Research & Politics, 2024; von der Heyde et al., 2025
2026
AI analysts meet real codebooks
The LongDA benchmark gave AI agents raw data files, long documentation and 505 analysis questions drawn from 17 public US national surveys, so that they had to find the key information in the documents before computing an answer. Its authors report "substantial performance gaps even among state-of-the-art models."
Credit
Yiyang Li and six co-authors, January 2026
Watch for
Coding with stated confidence
Google's Gemini Decision 1, introduced in November 2026, writes no text: it returns probabilities for up to 1,000 yes/no, choice or rubric questions about one input. Google says it "leads across the suite, with the lowest calibration error" on its benchmarks, meaning its stated probabilities track how often it is right, and answers in 150 to 250 milliseconds for inputs of up to 16,000 tokens.
Achieved whenan outside test on human-coded survey answers shows both accurate codes and calibrated probabilities.
Claimed by
Google, November 2026 (see the Note "A thousand questions, one pass")
Open
01
Code answers in five languages
A shared test set of written survey answers in at least five languages, each coded by trained people into a published code list, with scores for AI models against those codes and against a second human coder.
The briefCredit · unclaimed02
Score AI-built code lists
A test in which AI models receive only raw answers, with no code list, build their own categories and apply them, and are scored against the code list that trained human coders built for the same answers.
The briefCredit · unclaimed
Open 01 · the brief
Code answers in five languages
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A shared test set of written survey answers in at least five languages, each coded by trained people into a published code list, with scores for AI models against those codes and against a second human coder.
Why it is still open
The published tests disagree by language. Mellon and colleagues found AI close to human coders on English answers, and von der Heyde and colleagues found most models fell short on German ones; searches found no benchmark covering several languages.
Built with what exists
The British Election Study's open-text "Strings" files and its 50-category codes; coded open answers from other national election and social surveys; Gemini Decision 1 for probability outputs (see the Note "A thousand questions, one pass") and other current models; Python.
First step
Download the British Election Study files and write to four other national election studies asking for their coded open answers.
Done looks like
At least 2,000 answers in each of five languages, with accuracy and Cohen's kappa, a measure of agreement beyond chance, for each model, next to human-to-human agreement on the same answers.
Open 02 · the brief
Score AI-built code lists
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A test in which AI models receive only raw answers, with no code list, build their own categories and apply them, and are scored against the code list that trained human coders built for the same answers.
Why it is still open
The British Election Study codes its "most important issue" answers with trained human coders into 50 categories and 12 broad ones. The AI tests found either give the model the code list first, as Mellon and colleagues did, or compare themes from a single transcript, as the Southampton study did; none found compares AI-built code lists with a human one on survey answers.
Built with what exists
British Election Study strings and codes; current AI models; Python with standard measures of agreement between groupings, such as adjusted mutual information.
First step
Register on the British Election Study site, download one wave's strings and codes, and run one model.
Done looks like
On 10,000 answers, the share of human categories the AI list recovers, the share of AI categories with no human match, agreement between AI and human coding of the same answers, and blinded ratings of both lists by ten researchers.