Industries / Technology / Software and SaaS / AI and machine learning
Training
Training turns data and computing power into a working model: choosing the model's design, running it through the data until its errors shrink, and then teaching it to follow instructions.
7
Milestones achieved, from 1986 to 2025.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1986
Learning from errors
David Rumelhart, Geoffrey Hinton and Ronald Williams described back-propagation in Nature: a procedure that repeatedly adjusts the strengths of the connections in a network so that its output moves closer to the desired output. Hidden units, which are neither input nor output, came to represent useful features of the task on their own, which the paper says set the method apart from earlier, simpler ones such as the perceptron's.
Credit
David Rumelhart, Geoffrey Hinton and Ronald Williams
2017
Attention replaces recurrence
The Transformer, a model design from Google researchers, dropped recurrence (reading a sentence one word at a time) and relied only on attention, a way for each word to weigh every other word at once. Because the work could be spread across many chips in parallel, it trained quickly: the English-to-French model set a new single-model record after 3.5 days on eight GPUs, by the authors' count.
Credit
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin, Google Brain, Google Research and the University of Toronto
2021
Small add-ons for fine-tuning
Fine-tuning is further training of a finished model for one job. Edward Hu and colleagues at Microsoft proposed LoRA (low-rank adaptation): keep the model frozen and train two small added tables of numbers instead. Against full fine-tuning of the 175-billion-parameter GPT-3, they report 10,000 times fewer trained parameters, a third of the GPU memory, and no added delay when the model answers. It is now a standard method in Hugging Face's open PEFT library.
Credit
Edward Hu, Yelong Shen and colleagues, Microsoft
2022
More data, smaller model
In 2020 Jared Kaplan and colleagues at OpenAI found that a model's error falls as a power law with its size, its data and its compute, and advised spending a fixed budget mainly on very large models trained on relatively little data. In March 2022 Jordan Hoffmann and colleagues at DeepMind trained more than 400 models and found that size and training data should grow in equal proportion. Their 70-billion-parameter Chinchilla used the same compute as their 280-billion-parameter Gopher, with four times the data, and beat it and larger models such as GPT-3. In 2024 Tamay Besiroglu and colleagues at Epoch AI re-fitted one of Chinchilla's three methods, found it inconsistent, and got a result that agrees with the other two.
LessonThe first scaling rule left the largest models of the time "significantly undertrained", in DeepMind's words, so a rule fitted to experiments needs checking by more than one method.
Credit
Jordan Hoffmann and colleagues, DeepMind; Jared Kaplan and colleagues, OpenAI; Epoch AI for the replication
2022
Taught by human rankings
Long Ouyang and colleagues at OpenAI fine-tuned GPT-3 first on example answers written by labelers, then on labelers' rankings of its answers, using reinforcement learning from human feedback (RLHF), in which the model is rewarded for answers people rank higher. In their tests, people preferred the answers of the resulting 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, a model 100 times larger.
Credit
Long Ouyang, Jeff Wu, Xu Jiang and colleagues, OpenAI
2022
A logbook of failures
Meta AI trained OPT-175B, a 175-billion-parameter model, on 992 A100 GPUs and published the logbook of the run. Over two months, hardware failures caused at least 35 manual restarts and the swapping out of more than 100 machines, plus an estimated 70 or more automatic restarts. When the loss (the model's training error) suddenly diverged, the team lowered the learning rate and restarted from an earlier saved copy of the model.
LessonThe team wrote that cost estimates assuming "no hardware failures or training instabilities" understate what a model costs to make.
Credit
Susan Zhang, Stephen Roller, Naman Goyal and colleagues, Meta AI
2025
Reasoning learned by trial
DeepSeek trained R1-Zero by reinforcement learning on problems a program can mark right or wrong, such as math and coding, without human-written worked solutions, and the model learned habits such as checking its own work. Nature published the work after peer review on 17 September 2025. Its supplement puts the reasoning stage at 147,000 H800 GPU-hours, or $294,000 at an assumed rental price of $2 an hour, on top of the base model it started from.
LessonA cost figure for one stage is not the cost of a model, yet such figures move markets: when R1 came out on 27 January 2025, Nvidia lost close to $600 billion in market value in one day, CNBC reported, the largest one-day drop for a US company.
Credit
DeepSeek-AI
Watch for
Training for repeatable answers
Microsoft applied scientists say 50 steps of reinforcement learning on 24 H100 GPUs took the open Qwen3.8-27B model from passing 38 of 507 agent tasks on all 20 tries to passing 107. Tasks passed at least once fell from 453 to 446. No benchmark task was used in training, but the benchmark is Microsoft's own ThinkingBox, and the authors have not measured whether the gain carries over to other agent settings (see the Note "All twenty times").
Achieved whena group outside Microsoft repeats the gain on held-out tasks of a public benchmark.
Claimed by
Tuhin Kundu and Mirco Milletari, Microsoft, November 2026
Training without loss spikes
A loss spike is a sudden jump in training error that can force a restart. Moonshot AI says its MuonClip optimizer (the rule that decides how to change a model's numbers at each step), which adds a technique it calls QK-clip, kept the 15.5-trillion-token pre-training of its 1-trillion-parameter Kimi K2 free of any loss spike. A preprint from October 2026 by Masato Fujitake reports that Muon-family optimizers can still produce large spikes in sparse mixture-of-experts training; it does not test QK-clip.
Achieved whenanother lab publishes the full loss curve of a run of similar size that used the method and shows no spike.
Claimed by
Moonshot AI (Kimi Team), July 2025, revised February 2026
Open
01
Spike warnings on real runs
A labeled public set of loss spikes taken from large training runs whose logs are published, and a warning tool scored on it: how many steps before each spike it raises an alarm, and how often it raises a false one.
The briefCredit · unclaimed02
Repeatable answers, checked outside
An open repeat of Microsoft's ThinkingBox result: train a small open model by reinforcement learning to pass the same task every time, then measure it on a public benchmark it never saw in training.
The briefCredit · unclaimed
Open 01 · the brief
Spike warnings on real runs
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A labeled public set of loss spikes taken from large training runs whose logs are published, and a warning tool scored on it: how many steps before each spike it raises an alarm, and how often it raises a false one.
Why it is still open
Teams still handle spikes by hand. The OPT-175B team made at least 35 manual restarts and rolled back to earlier copies of the model when the loss diverged. Open tools such as trainspotter (2026) locate spikes in a log once they have happened, and its authors call its checks heuristics. Searches of arXiv and GitHub found papers on why spikes happen and a 2023 MosaicML prediction repository with no code or description, but no public test of advance warning on real large runs.
Built with what exists
the Allen Institute for AI's published Weights & Biases training logs for its open OLMo 2 models (1, 7 and 13 billion parameters); the OPT-175B logbook; Python with pandas; simple statistics or a small sequence model over the loss and the other measures each log records.
First step
Download the OLMo 2 7B training log and mark every step where the loss rises more than a set amount above its recent average.
Done looks like
on runs held out from tuning, the tool warns at least 100 steps before at least half of the labeled spikes, with no more than one false alarm per 10,000 steps, and the labels, code and scores are public.
Open 02 · the brief
Repeatable answers, checked outside
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
An open repeat of Microsoft's ThinkingBox result: train a small open model by reinforcement learning to pass the same task every time, then measure it on a public benchmark it never saw in training.
Why it is still open
Microsoft's November 2026 result was measured on its own benchmark, its authors have not measured transfer to other agent settings, and no outside repeat was found. τ-bench's authors defined pass^k, the share of tasks passed on all k tries, in 2024, and found GPT-4o passed all eight tries on fewer than 25% of retail tasks.
Built with what exists
τ²-bench, Sierra's open follow-up to τ-bench, with customer-service tasks an agent completes through tools; an open-weight model of about 8 billion parameters; the verl or Hugging Face TRL reinforcement-learning libraries; one rented machine with eight H100 GPUs.
First step
Run the untrained model eight times on each τ²-bench retail task and record its pass^1 and pass^8.
Done looks like
pass^8 before and after training on τ²-bench tasks held out from training, from three training runs with different random seeds, with the share of tasks passed at least once also reported, and code and logs public.