Industries / Technology / Software and SaaS / AI and machine learning
Data
Data work gathers, cleans, labels and licenses the text, images and other material that models learn from.
7
Milestones achieved, from 2007 to 2025.
3
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
2007
A free web archive
Gil Elbaz set up Common Crawl, a nonprofit that downloads billions of web pages about once a month and gives the archive away; it now holds more than 10 PiB. A Mozilla Foundation review of 47 text models released from 2019 to October 2023 found that at least 30 of them (64%) were trained on a filtered version of it, and that it made up more than 80% of the tokens (word pieces) GPT-3 learned from.
LessonCommon Crawl keeps hate speech on purpose so researchers can study it, so the job of filtering falls to each model builder.
Credit
Gil Elbaz, Common Crawl Foundation; review by Stefan Baack, Mozilla Foundation
2009
Crowd-checked labels at scale
ImageNet paid online workers on Amazon Mechanical Turk, a marketplace for small paid tasks, to confirm whether each candidate photo showed its category. By August 2014 it held 14,197,122 labeled images in 21,841 categories, and its makers counted 5 errors in 1,500 test images they rechecked.
LessonA 2021 outside check found at least 6% of ImageNet's validation labels were wrong, so even crowd-checked labels need a second audit.
Credit
Jia Deng, Li Fei-Fei and colleagues
2023
The people behind the labels
TIME reported that Kenyan workers at Sama, an outsourcing firm, labeled tens of thousands of passages describing abuse, murder and torture so OpenAI could train a filter for ChatGPT. TIME put their take-home pay at about $1.32 to $2 an hour, against $12.50 an hour that OpenAI paid Sama; Sama disputed the figures and ended the work in February 2022, eight months early.
LessonThe labeling that makes a chatbot safer can expose the labelers to the harm it filters.
Credit
reported by Billy Perrigo, TIME
2023
Abuse images in a dataset
The Stanford Internet Observatory found "many hundreds" of known child sexual abuse images in LAION-5B, an open list of billions of image links used to train image generators. LAION took the list down and in August 2024 released Re-LAION-5B, with 2,236 links removed after matching against hash lists (fingerprints of known abuse images) from the Internet Watch Foundation and the Canadian Centre for Child Protection.
LessonCheck a dataset against known-abuse hash lists before release, not after.
Credit
David Thiel, Stanford Internet Observatory; LAION
2024
Data by license
In February 2024 Google announced access to Reddit's Data API so it could "display, train on, and otherwise use" Reddit content. In July 2024 the Data Provenance Initiative reported that in one year, new robots.txt rules (a file that tells crawlers which pages to skip) had fully closed at least 5% of the tokens in C4, a widely used web dataset, and at least 28% of its most actively maintained sources.
LessonAs sites close to free crawling, access to good data moves to paid deals.
Credit
Google and Reddit; Shayne Longpre and the Data Provenance Initiative
2024
Copies of copies degrade
A paper in Nature showed that training models "indiscriminately" on text made by earlier models causes "irreversible defects": rare content drops out first. In August 2026 Pew Research Center ran an AI detector over 10,000 English pages from Common Crawl's July 2026 crawl and found that 10% of all pages, and over a third of pages published after ChatGPT's release, showed signs of AI authorship.
LessonAs the web fills with model output, builders must tell human text from machine text before training.
Credit
Ilia Shumailov, Yarin Gal and colleagues; Pew Research Center
2025
A required list of sources
Since 2 August 2025, the EU AI Act has required anyone who sells a general-purpose AI model in the EU to publish a summary of its training content on a Commission template, including the top 10% of web domains scraped; enforcement began on 2 August 2026. Five days later, the AI Accountability Lab at Trinity College Dublin had found 39 summaries, counted 20 models without one, and reported that no provider named the specific websites it crawls.
LessonA required form without checks produced vague answers.
Credit
European Commission AI Office; AIAL
Watch for
Paid when content is used
Cloudflare, a network company that sits in front of many websites, says it is replacing its 2025 "pay per crawl" beta with "pay per use", where AI firms pay publishers when their pages appear in answers, starting with Ceramic.ai and You.com. It also said its default settings would block "mixed-use" crawlers on ad-supported pages from 15 September 2026 for new and free customers.
Achieved whenCloudflare or an outside party publishes how much publishers have been paid, and by how many AI companies.
Claimed by
Cloudflare, July 2026
Models from licensed text only
A team led by Nikhil Kandpal, Brian Lester and Colin Raffel released the Common Pile, 8 TB of public-domain and openly licensed text from 30 sources, and says two 7-billion-parameter models trained on it perform about as well as Llama 1 and Llama 2 7B, which were trained on unlicensed text with similar computing budgets.
Achieved whenan outside group retrains or re-tests the models and confirms the match.
Claimed by
Kandpal and 26 co-authors, June 2025
Nearly a third AI-written
Using the Pangram detector, a team reports that 27.5% of tokens in June 2026 web data were AI-generated, and 31.1% by August, and that for models trained on large amounts of human text, adding AI tokens raises the error almost at once.
Achieved whenanother team, with a different detector, reproduces the share and the training effect.
Claimed by
Jenna Russell, Mohit Iyyer and five co-authors, September 2026
Open
01
Canary count in open datasets
A public count of "canary strings" left in the main open training datasets. A canary is a unique code that test makers put in their files so model builders can find and remove test material before training.
The briefCredit · unclaimed02
Crawler promises, checked
Twenty small test websites, each with robots.txt rules aimed at named AI crawlers, logging every visit for 90 days. Each visit is matched against the crawler company's published IP address list, and each company's measured behavior is compared with what its EU training summary says it does.
The briefCredit · unclaimed
Open 01 · the brief
Canary count in open datasets
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public count of "canary strings" left in the main open training datasets. A canary is a unique code that test makers put in their files so model builders can find and remove test material before training.
Why it is still open
The Alignment Research Center's 2022 canary page says such documents "should never appear in training corpora", and a 2024 Stanford survey found that only 9 of 30 model developers report how much their training data overlaps with test sets. No published count of canaries inside open datasets was found.
Built with what exists
infini-gram's free count API, which indexes DCLM-baseline, Dolma v1.7, OLMo-mix-1124, RedPajama, the Pile and C4; Hugging Face's dataset streaming plus ripgrep for FineWeb, which infini-gram does not index; canary strings copied from the repositories of BIG-bench and other test sets.
First step
Collect 10 canary strings from test-set repositories and run infini-gram count queries for each.
Done looks like
A public table of at least 10 canary strings across at least 6 datasets, with hit counts, sample document links, 50 hits checked by hand, and a script anyone can rerun on a new dataset release.
Open 02 · the brief
Crawler promises, checked
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
Twenty small test websites, each with robots.txt rules aimed at named AI crawlers, logging every visit for 90 days. Each visit is matched against the crawler company's published IP address list, and each company's measured behavior is compared with what its EU training summary says it does.
Why it is still open
The AI Accountability Lab found that no provider names the sites it crawls, and ActuIA found that 21 of 24 summaries name no site in the required field; DeepSeek says its robot respects robots.txt, and nobody has checked. A 2025 study of 130 bots found that bots obey strict robots.txt rules less often and that AI search crawlers rarely read the file, and goodbot-badbot.com monitors a single site, but neither ties results to each company's written promise.
Built with what exists
cheap domains on a small server running nginx or Caddy with full access logs; published crawler IP lists such as OpenAI's gptbot.json; Web Bot Auth signatures where crawlers send them; the AI Accountability Lab's assessments of each summary.
First step
List the companies whose EU summaries or crawler pages name a crawler and promise to honor robots.txt, and put up the first three test sites.
Done looks like
90 days of logs from 20 sites; for each crawler company, the share of IP-verified visits that obeyed the rules; a table setting that share beside the company's written claim; raw logs published with personal data removed.