Microsoft ThinkingBox
All twenty times
By Microsoft's own count, 50 steps of training took an AI model from passing 38 of 507 test tasks on all 20 tries to passing 107.
Agents · Checked 13 Nov 2026 · 12 sources · 10 min

A request reaches a support desk: move a hotel booking from a suite to a deluxe room. Company policy allows that change only with a manager's approval. Any clerk can learn the rule in a minute. What the company needs from that clerk takes longer to find out: that the rule is followed on the fortieth request the way it was on the first.
That request is also an exam question for software. It opens a research write-up from Microsoft's Command Line publication, posted on 6 November 2026, the second of the company's write-ups about a testing system it calls ThinkingBox. In it the booking is worth $1,450. The one sitting the exam is an AI agent: a model of the kind behind chatbots that has also been given tools, so that besides writing a reply it can look up a booking and change it. The model in this case, Qwen3.8-27B, is open-weight, meaning outsiders can download it and train it further. It was given the request 20 separate times, and by Microsoft's account it made the change anyway in 15 of the 20.
ThinkingBox is the system that produced that count. As the first write-up described it on 19 August 2026, each attempt runs inside a fresh, sealed copy of a fictional company's records, and the agent's tools are the only way to change them. A simulated user plays the person asking and supplies missing details only when asked. When the agent stops, a program inspects the records it left behind, and if they are as they should be the attempt passes, by whatever route the agent took. There are 507 tasks in all, a fixed set of the kind called a benchmark. For 477 of them the records alone decide, and the other 30 add a question answered by another AI model acting as judge.
For those 477, nothing the agent says counts, and the second write-up's tally of failures shows what a grader going on the conversation would miss. Across 12 models it found 79,853 failures in 121,680 valid attempts, and in 67.24% of them, about two in every three, the agent still ended with a reply in the style of a completed job, a use of a tool that changed a record, and no explicit error. The paper behind both write-ups, One Success Isn't Reliability, concludes that an agent's replies and its use of tools are weak signs of whether a task was completed.
Each task is run 20 times, which for one model makes 507 times 20, or 10,140 attempts, and the runs do not all end the same way. Models of this kind are normally run with some chance in their choice of words, so one request can leave different records on different runs. Those attempts have to be boiled down to a score, and there is more than one way to do it.
Suppose a made-up desk that gets ten kinds of request, with an agent tried 20 times on each, 200 attempts in all. Say one agent gets five kinds right every time and the other five wrong every time. That is 100 passes in 200 attempts, or 50%. Say a second agent gets every kind right on half its tries. That is also 100 passes, and also 50%. A manager could hand the first agent its five kinds of request. The second can't be left alone with any, though it has shown it can do all ten.
So there are three things to measure: the share of all attempts that pass, the share of tasks passed at least once, and the share passed every time. The first made-up agent scores 50%, five kinds in ten and five in ten. The second scores 50%, ten in ten and none. The first write-up labels the three pass@1, pass@20 and pass^20, the third a kind of measure that the 2024 paper on a benchmark called τ-bench introduced.
Before any training, Qwen3.8-27B passed 51.70% of its attempts. It passed 453 of the 507 tasks at least once, and 38 of them on all 20 attempts. That is an agent much nearer the second made-up one than the first. The Microsoft authors ran these measurements, like every ThinkingBox measurement before and after them, and no independent replication has been reported. The company that built the exam also chose the questions, ran the model and marked the papers. Research is usually first reported that way, and it leaves each figure as one group's measurement until someone outside repeats it.
The grade becomes the reward
The second write-up is about another use for the program that marks the exam. A model is, underneath, a very long list of numbers, and training means adjusting them. In the kind of training called reinforcement learning, the model attempts a task, the attempt is given a score, and the numbers are nudged so that whatever earned a high score becomes more likely next time. Training like that needs something to do the scoring, and ThinkingBox already had it. Microsoft made the pass or fail check the reward: 1 for a pass, 0 for a fail.
The method is called group relative policy optimization, and the DeepSeekMath paper introduced it. In general terms, a method of this kind learns by comparison: the model attempts one task several times, and its numbers shift toward what the better-scoring attempts in that group did and away from the rest, so a group whose attempts all pass, or all fail, offers nothing to compare. Before training, the Microsoft researchers tried the untrained model on each candidate task and dropped those where its estimated chance of success was below 0.08 or at 0.90 and above. What remained were tasks it was estimated to pass some of the time.
The run lasted 50 steps, and each step used 18 groups of 8 attempts. That is 144 attempts a step and 7,200 in all, fewer than the 10,140 it takes to sit the exam once. The 187 tasks it trained on are not public and are not in the benchmark, though the authors state that the two sets share a framework, a user design, a grading method and a task style. The exam resembled the lessons, and whether the gain carries over to unrelated settings has not been measured.
After the 50 steps, the three measures had not moved together. The share of attempts passed rose from 51.70% to 60.78%. The tasks passed at least once went from 453 to 446, slightly down. The tasks passed on all 20 attempts went from 38 to 107.
As a share of the 507 tasks, that last change is from 7.50% to 21.10%, about one task in five where there had been fewer than one in ten. The authors describe a model that became more repeatable without becoming broader. The set of tasks it could do at all did not grow, and it failed less often at the ones it could.
On the hotel request, by Microsoft's account, the trained model left the booking and its payments untouched in all 20 attempts, and every expected record matched in 19. The authors add that leaving the booking alone is necessary but does not by itself pass the task. The every-time measure is stricter again, since one miss in 20 keeps a task out of the 107.
Which model looks best depends on which measure is read. In the second write-up's table, the trained model passes 60.78% of its attempts, ahead of a model called GPT-6 Astra at 58.31%. On tasks passed every time the order turns over, with 107 for the trained model and 231 for the other.
ThinkingBox, from Microsoft's two write-ups
Ahead on attempts passed, behind on tasks passed every time
There are 507 tasks in all. Tasks passed at least once went from 453 to 446 over the same training.
| Before training | After the 50 steps | GPT-6 Astra | |
|---|---|---|---|
| Share of attempts passed | 51.70% | 60.78% | 58.31% |
| Tasks passed on all 20 attempts | 38 | 107 | 231 |
Attempt 21
Twenty passes in 20 attempts guarantee nothing about the next one. Take the made-up desk again, and suppose a third agent that gets each kind of request right 9 times in 10, with every attempt a fresh try unaffected by the last. Its chance of passing a request twice in a row is 0.9 times 0.9, or 0.81, and multiplying on to 20 in a row leaves about 0.12, roughly one chance in eight. So this made-up agent could show a perfect record on one kind of request and go on failing it one time in ten. On most kinds it would miss at least once and score nothing on the every-time measure.
The passes are also passes inside a simulation. The authors state that the tasks are synthetic reconstructions with simulated users, not production traffic, the real requests a working business receives. The paper adds that an attempt can pass while the agent misreports the outcome to the user, and that one AI model, GPT-5.4-mini, from the same model family as one of the agents being tested, played the user for every model. Its Appendix C.4 names a different model, GPT-5 Chat, for all runs, while the second write-up assigns GPT-5 Chat to training only.
What Microsoft released is research code, with no product attached. The testing system and the training code are public on the code-sharing site GitHub, but no source says the trained model has been released, and the second write-up says no provider offers it as a service. The store of data that the OpenEnv documentation for ThinkingBox names as the source of the runnable benchmark returned a not-found error on 13 November 2026. And a PyTorchKR forum post of 27 October 2026, which describes itself as a GPT-assisted draft, reported one training run with no variation of the random seed, the starting number behind a run's chance choices. It also found differing length limits and parsers, the code that reads a model's output, across the paper, the code and the blog.
If the figures hold outside Microsoft's benchmark, a team with a program that checks the final state of its own records has, in that program, a reward for this kind of training. The reference run used three machines of eight H100 GPUs each, 24 of the processors AI models are trained on. The one measure that fell, tasks passed at least once, from 453 to 446, matches a 2025 finding by Yue and colleagues that models trained by reinforcement learning beat the models they started from when few attempts are allowed, while the starting models solve more problems at least once when many are allowed. The second write-up links that paper without drawing the comparison.
Check it yourself
- Open the second write-up and find Qwen3.8-27B in the model table, before and after training. It confirms the counts of 38 and 107 tasks passed on all 20 attempts, and the pass@1 figures of 60.78% for the trained model and 58.31% for GPT-6 Astra.
- Open the Hugging Face training post and read its results table. It gives 5,242 and 6,163 passing attempts and the fall from 453 to 446 tasks passed at least once. Divide each of the first two by the 10,140 attempts a model makes to get 51.70% and 60.78%.
- Read the definitions in the first write-up to confirm what pass@1, pass@20 and pass^20 each count, and that an attempt is graded on the records it leaves.
- Read Appendix C.7 and Table 15 in the paper's HTML version. Table 15 gives the plain counts of tasks passed 20 times in 20. C.7 gives the formula behind the paper's own pass^20 column, each task's observed success rate raised to the 20th power and then averaged, which is why that column shows 10.93% for the untrained Qwen3.8-27B where 38 of 507 is 7.50%.
- Follow the OpenEnv documentation for ThinkingBox to the store of benchmark data it names, a repository that returned a not-found error on 13 November 2026. Whether its pinned thinkingbox-bench-v1.0 release loads decides whether the standard benchmark runs can be repeated.
- Download microsoft/thinkingbox-training and run the dry-run command from its README. It prints the launch command for the 50-step configuration without allocating any GPUs.
- Install microsoft/thinkingbox on Linux or WSL and run its bundled cloud_drive install check against a model endpoint of your own. It confirms that the testing system runs.
Sources
- The check that catches agent failures can also train them. Command Line, Microsoft, 6 November 2026.
- How we built ThinkingBox to measure whether agents finish the job. Command Line, Microsoft, 19 August 2026.
- One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows. arXiv, HTML version 5, 31 October 2026.
- One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows. arXiv, 20 August 2026.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv, 17 June 2024.
- Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox. Hugging Face blog, Microsoft and Hugging Face, 22 October 2026.
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv, 5 February 2024.
- microsoft/thinkingbox. GitHub.
- microsoft/thinkingbox-training. GitHub.
- ThinkingBox. OpenEnv documentation, Hugging Face.
- ThinkingBox-Training 공개: DB 상태 채점 시 강화 학습 보상으로 매번 성공하는 과제 수를 2.8배 ↑. PyTorchKR, 27 October 2026.
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. arXiv, 18 April 2025.