Industries / Finance and business services / Translation services
Checking quality
Checking quality is the work of judging whether a translation is accurate and fit for its purpose, by a second linguist, an examination, or a score assigned by a person or a program.
7
Milestones achieved, from 1966 to 2022.
1
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1966
Readers rate two qualities
For the ALPAC review, the psychologist John B. Carroll ran an experiment that scored human and machine translations of Russian scientific text. ALPAC drew on it to conclude that unedited machine output "is sometimes misleading and sometimes wrong" and makes "slow and painful reading", and its cost study assumed readers took twice as long over such output. The report's method judged translations on intelligibility, how easily they are understood, and fidelity, how much of the source's meaning they keep; the US ARPA machine translation program of the early 1990s later scored adequacy, fluency and comprehension in a similar way.
Credit
John B. Carroll, Harvard University
2002
A score without judges
Researchers at IBM published BLEU, a score that compares a machine translation with one or more human translations of the same text by counting the runs of words they share. They described it as "quick, inexpensive, and language-independent" and as an "automated understudy to skilled human judges", since human evaluations took weeks or months and could not be reused.
Credit
Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu, IBM T. J. Watson Research Center
2014
A shared list of errors
Multidimensional Quality Metrics (MQM) gave buyers and providers of translation a shared vocabulary of "issue types", so that a client can declare which kinds of error count and how heavily for a job, and human and machine translations can be scored the same way. It came out of the EU-funded QTLaunchPad project. The MQM Council has maintained it since 2016 under a CC BY 4.0 license and aligns it with ISO 5060:2024, ISO's guidance on scoring translations by error types and penalty points.
Credit
Arle Lommel, Hans Uszkoreit and Aljoscha Burchardt, QTLaunchPad
2015
A second linguist required
ISO 17100:2015, the international requirements for translation services, replaced the European standard EN 15038 of 2006. It requires a second person to revise every translation, the "compulsory second pair of eyes" in the words of the UK's Association of Translation Companies. In the revision of the standard under way since 2025, one debate is whether editing machine output should count as that revision.
Credit
ISO technical committee 37 (ISO/TC 37)
2018
A parity claim retested
In March 2018 Microsoft researchers reported that their Chinese-to-English news system was "at human parity" with professional translators, meaning raters of single sentences found no statistically significant difference. Samuel Läubli, Rico Sennrich and Martin Volk had professional translators compare the same human and machine translations both as isolated sentences and as whole documents; the raters preferred the human translation more strongly when they saw whole documents.
LessonSome errors show only across sentences, so judging sentences one at a time can make a machine look as good as a person.
Credit
Läubli, Sennrich and Volk, University of Zurich and University of Edinburgh; Hany Hassan and colleagues, Microsoft, for the original claim
2021
Experts versus crowd raters
Google researchers had professional translators mark every error, under MQM and with whole documents in view, in the top outputs of the 2020 WMT shared task for English to German and Chinese to English. Their ranking differed substantially from the official one built from crowd workers' scores: the professionals ranked every human translation first, while crowd workers had underrated human translations and favored literal ones. MQM cost about three times as much as simpler ratings.
LessonOnce machine output reads fluently, untrained raters can rank systems wrongly.
Credit
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan and Wolfgang Macherey, Google Research
2022
Learned scores beat BLEU
The WMT 2022 metrics task judged automatic scores against MQM ratings by experts across news, social media, e-commerce and chat text. Scores trained on human judgments, led by Google's MetricX XXL and Unbabel's COMET-22, ranked first and second; BLEU ranked 19th of 20 primary submissions, and the organizers titled their report "Stop Using BLEU". By 2026 the task, merged with quality estimation, found that AI models prompted to act as judges, with no task-specific training, beat every traditional and trained score. Its new test of picking out translations fit to publish without a human check was hard for all: of the translations the best entry, built on GPT-5.4, cleared, 44% were free of errors by the human judgment.
Credit
Markus Freitag, Ricardo Rei and the WMT metrics task organizers; Alon Lavie and the WMT26 organizers
Watch for
Gates that decide publication
Pangeanic says its MTQE v2 system, which scores a machine translation without a human reference and then publishes it, repairs it automatically or sends it to a person, let through only 65 of 6,006 known-bad translations (1.1%) in 16 language pairs of the ACES test set, and repaired 67.6% of them automatically. Its own whitepaper notes that the test used only incorrect translations, so it "cannot report false rejection", the share of good translations sent to review needlessly.
Achieved whena party outside Pangeanic tests the gate on a realistic mix of good and bad translations rated by professionals and publishes both the share of bad translations it passes and the share of good ones it blocks.
Claimed by
Pangeanic, August 2026
Open
01
Test the gates buyers use
An outside test of the commercial gates that decide whether a machine translation is published without a human check, run at their default settings on the WMT 2026 test of that decision, whose translations professionals have already judged. For each gate, report how many translations with errors it would publish and how many clean ones it would hold back, beside the shared task's own results.
The briefCredit · unclaimed02
Second linguist or AI reviewer
A head-to-head test of professional revisers and AI reviewers on the same translated documents, with errors planted at known places: wrong numbers, dropped negations, swapped terms, missing sentences, and errors that show only across sentences, such as a term translated two ways. Measure the share of planted errors each catches and how often each flags a correct passage.
The briefCredit · unclaimed
Open 01 · the brief
Test the gates buyers use
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
An outside test of the commercial gates that decide whether a machine translation is published without a human check, run at their default settings on the WMT 2026 test of that decision, whose translations professionals have already judged. For each gate, report how many translations with errors it would publish and how many clean ones it would hold back, beside the shared task's own results.
Why it is still open
WMT 2026 ran the first public test of the decision, with 18 entries; Phrase, a translation-software company, was the only maker of translation tools among the entrants, and the best entry's clearances were right 44% of the time. Gates sold to translation buyers, such as Pangeanic's and TAUS's, were not entered. Pangeanic's August 2026 test of its own gate used only bad translations, and Gladkoff and colleagues (ICNLSP, September 2026), testing TAUS EPIC on MQM-annotated EU translations, concluded that segment scores "should not be used as a standalone basis for routing, release, or review bypass".
Built with what exists
The WMT 2026 evaluation set, human labels and scoring code, released on GitHub (wmt-conference/wmt26-mteval); commercial quality-estimation services whose terms allow results to be published; Unbabel's open COMETKiwi model as a baseline; Python.
First step
Download the WMT 2026 evaluation set, rerun one of the organizers' baselines to match the paper's figure, then run one commercial gate on one language pair.
Done looks like
At least 5 commercial gates on the shared task's 8 priority language pairs; for each, the Matthews correlation computed as the shared task computed it, the share of translations with errors it passes and the share of error-free ones it holds back, with 95% confidence intervals; code and every decision published.
Open 02 · the brief
Second linguist or AI reviewer
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A head-to-head test of professional revisers and AI reviewers on the same translated documents, with errors planted at known places: wrong numbers, dropped negations, swapped terms, missing sentences, and errors that show only across sentences, such as a term translated two ways. Measure the share of planted errors each catches and how often each flags a correct passage.
Why it is still open
ISO 17100 makes a second-person revision mandatory, and revisers' error detection has been studied: Ipsen and Dam (2016) scored nine revisers and found that the best started from the translation rather than the source text. Google researchers' MQM re-annotation method (October 2025) has raters review earlier error annotations, including machine ones, and reports that this catches errors missed in the first pass. A localization specialist's GitHub test of October 2026 compared one AI judge with the author's own review of 60 English-to-Russian sentences: they agreed on pass or fail 90% of the time, but the judge failed only 3 of the 21 sentences the author failed. This research found no published test that plants the same errors in whole documents and compares professional revisers with current AI reviewers.
Built with what exists
Openly published test documents with professional translations, such as the WMT25 test sets and references; a short script that plants errors and records their positions; AI models as reviewers, such as Claude Fable 5.2 (see the Note "Three working devices"), Gemini and Mistral Large 4 (see the Note "A trillion open parameters"); MQM error categories for scoring.
First step
Take five English-to-Italian documents from the WMT25 test set with their human translations, plant 10 errors in each, and give them to one professional reviser and one AI model.
Done looks like
40 documents in 2 language pairs with 400 planted errors, reviewed by at least 6 professional revisers and 3 AI models; the share of planted errors each catches, by error type, with 95% confidence intervals, and false flags per 1,000 words; documents, planted errors and results published.