Industries / Media, arts and recreation / Media and entertainment / Video games
Testing
Testers, called quality assurance (QA) in the trade, play unfinished versions of a game to find bugs, write them up so programmers can reproduce them, and check the game against the rules of each platform it will be sold on.
7
Milestones achieved, from 2009 to 2025.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
2009
Testers get their own company
Nintendo's Mario Club team in Japan debugs and play-tests Nintendo's games and reports back on them. In 2009 Nintendo split it off as a separate company, Mario Club Ltd., with 250 staff. By a Nihon Keizai Shimbun report relayed by MCV, the aim was to "improve treatment of part-time and contract employees": the part-timers who made up most of the staff would become contract employees, and contract employees would become full-time staff.
Credit
Nintendo
2020
Learning bots test shooters
Most automated game testing uses scripted bots that follow set routes. At the 2020 IEEE Conference on Games, Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar and Linus Gisslén showed that agents trained by deep reinforcement learning, which learn by trial and error toward a reward, could widen test coverage, find exploits and test map difficulty in first-person shooter levels. In 2023 a team at Electronic Arts and its SEED research group reported adding such agents to its scripted-bot testing for games including Battlefield 2042 and Dead Space (2023); by its count, Battlefield V alone had 601 features that would take about 0.5 million hours to test by hand.
LessonResearch methods were slow to reach production because, the EA team wrote, "the development environment can be very different from the final product.".
Credit
Joakim Bergdahl, Linus Gisslén and colleagues, SEED and Electronic Arts
2020
A broken launch is pulled
Cyberpunk 2077 was released in December 2020 and drew heavy criticism for bugs and poor performance on the base PlayStation 4 and Xbox One. CD Projekt Red apologized "for not showing the game on base last-gen consoles before it premiered" and said it "should have paid more attention to make it play better" on them. On 17 December 2020 Sony said it would offer full refunds and remove the game from its PlayStation Store "until further notice," and Microsoft said it would refund all digital buyers.
LessonA game sold for older hardware has to be tested and shown on that hardware; when it was not, a platform owner refunded buyers and took the game off sale.
Credit
CD Projekt Red; Sony Interactive Entertainment; Microsoft
2021
Bugs counted from patch notes
Andrew Truelove and Iftekhar Ahmed of the University of California, Irvine and Eduardo Santana de Almeida of the Federal University of Bahia read 723 updates to 30 popular Steam games and sorted the 12,122 bug fixes they listed. The most common were information bugs, where the game does not show the player correct information about its world; crash bugs recurred most often and were treated as the most severe. Developers they surveyed tied bugs that persist to difficulty in testing, code quality and reproducing the bug.
Credit
Andrew Truelove, Eduardo Santana de Almeida and Iftekhar Ahmed
2022
Testers vote for a union
On 23 May 2022, 28 QA testers at Raven Software, a studio owned by Activision Blizzard, voted 19 to 3 to join the Communications Workers of America (CWA) as the Game Workers Alliance; NPR called it a first at a major video game company. Activision had argued that the testers alone could not form a bargaining unit, and the US National Labor Relations Board let the vote go ahead. The testers had struck earlier over layoffs and cited low pay and long hours before launches.
Credit
Raven Software QA workers; CWA
2025
AI graded on spotting glitches
Vision-language models are AI models that read images or video together with text. GlitchBench, presented at the CVPR 2024 conference, found the best model then, GPT-4V, averaged 43.4% at spotting and explaining odd or broken scenes from games. VideoGameQA-Bench, presented at NeurIPS 2025, tested 16 models on 2,236 images and 1,200 videos from more than 800 games: the best scored 82.8% at spotting glitches in images, and the authors found models "can generate useful bug reports for up to 50% of real-world glitches." Visual regression testing, checking that a new build still looks as it should, "remains one of the most challenging tasks."
Credit
Mohammad Reza Taesiri, Cor-Paul Bezemer and colleagues
2025
A contract for testers
ZeniMax Workers United-CWA, which represents more than 300 QA workers in Maryland and Texas at ZeniMax, a Microsoft-owned game company, formed in 2023 as the first video game studio union at Microsoft. In 2025 its members ratified a first contract "after over two years of bargaining," with "significant across-the-board wage increases, position upgrades, job security measures," by the union's account.
Credit
ZeniMax Workers United-CWA; Microsoft
Watch for
AI for most testing work
Square Enix set a target to "automate 70% of QA and debugging tasks in game development by the end of 2027" using generative AI, in a progress report on its medium-term business plan. The work is a joint project with the Matsuo-Iwasawa Laboratory of the University of Tokyo, titled "Joint Development of Game QA Automation Technology Using Generative AI."
Achieved whenSquare Enix reports the share of QA and debugging work automated and how it was measured, and an outside party, such as a published paper or the press checking with its testers, confirms it.
Claimed by
Square Enix, November 2025
Bots that retest a goalie
Electronic Arts researchers say their method, Reward-Adaptive Iterative Discovery, trains a group of reinforcement learning agents to score against the goalie AI of a development version of EA Sports NHL 26. In a single experiment it found six scoring exploits "qualitatively similar to those that playtesters had found in hours-long manual testing sessions."
Achieved whenEA reports how many exploits it found before a game's release and how many players later found, or another studio reproduces the method on its own game.
Claimed by
Florian Fuchs, Linus Gisslén and colleagues, Electronic Arts, July 2026
Open
01
Agents against testers in 3D
A public head-to-head between AI play agents and human testers in real 3D games. Take two open-source 3D games, such as SuperTuxKart and one of Godot's 3D demo projects, and plant 60 bugs of common kinds behind a switch: holes in the floor, stuck states, crashes, wrong text and broken event triggers. Give an AI agent that sees the screen and sends controller input, a reinforcement learning coverage bot and five paid human testers the same number of hours, and count what each finds.
The briefCredit · unclaimed02
Pre-check games for handhelds
A tool a small studio can run before asking Valve to rate its game for the Steam Deck. It plays the opening of the game through controller input, measures the smallest text in screenshots against Valve's rule that no character fall below 9 pixels high at 1280 by 800, logs the frame rate against Valve's 30 frames per second at 800p, and checks that launchers and text entry work with the controller. Its verdicts are then compared with Valve's published ratings.
The briefCredit · unclaimed
Open 01 · the brief
Agents against testers in 3D
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public head-to-head between AI play agents and human testers in real 3D games. Take two open-source 3D games, such as SuperTuxKart and one of Godot's 3D demo projects, and plant 60 bugs of common kinds behind a switch: holes in the floor, stuck states, crashes, wrong text and broken event triggers. Give an AI agent that sees the screen and sends controller input, a reinforcement learning coverage bot and five paid human testers the same number of hours, and count what each finds.
Why it is still open
GBQA, an ICLR 2026 workshop paper, planted 124 checked bugs in 30 games that AI agents had made, and the best model found 48.39%, but it reports no comparison with people. TITAN's comparison with human testers covered nine planted bugs in two closed commercial online games, with the humans held to five attempts. World of Bugs (2022) collects common 3D game bugs to train detectors, not to compare testers.
Built with what exists
The SuperTuxKart and Godot source code; an AI model that reads screenshots, such as Claude Fable 5.2 (see the Note "Three working devices"), wired to a virtual controller; Stable-Baselines3 for the coverage bot; OBS Studio to record every session; Python.
First step
Plant five bugs behind a switch in SuperTuxKart, such as a gap in one track's collision, and run one agent and one person on it for an hour each.
Done looks like
60 planted bugs across two games; 20 hours per game for the agent, the bot and each human tester; bugs found, bugs found per hour and false reports for each, with the bug list, code and recordings published.
Open 02 · the brief
Pre-check games for handhelds
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A tool a small studio can run before asking Valve to rate its game for the Steam Deck. It plays the opening of the game through controller input, measures the smallest text in screenshots against Valve's rule that no character fall below 9 pixels high at 1280 by 800, logs the frame rate against Valve's 30 frames per second at 800p, and checks that launchers and text entry work with the controller. Its verdicts are then compared with Valve's published ratings.
Why it is still open
Valve does the review itself. Developers request it, and can ask for a new review after an update; Valve's page lists the checks but offers no tool to run them in advance. Developers describe checking by hand: Brendon of Blendo Games wrote in February 2026 that his "caveman" method is to take a screenshot at 1280 by 800 and count how many pixels tall the text is. Searches of arXiv, GitHub and the web found no research on predicting the ratings and no public pre-check tool.
Built with what exists
A Steam Deck in developer mode; MangoHud, an open-source overlay that logs frame rates on Linux; Tesseract or PaddleOCR to find text and measure its height; a vision-language model to move through menus; the Steam Deck ratings shown on Steam store pages as the answers to check against; Python.
First step
Measure the smallest text in 10 owned games by hand and by OCR at 1280 by 800, and compare both with each game's Valve rating.
Done looks like
100 owned games with Valve ratings; the tool's Verified, Playable or Unsupported verdict agrees with Valve's on at least 85 of them; on 20 games, its smallest-text heights fall within 1 pixel of measurements made by hand.