Industries / Technology / Social media
Moderation
Moderation is the work of deciding which posts and accounts break a platform's rules or the law, and then removing, labeling or limiting them.
7
Milestones achieved, from 1996 to 2023.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1996
Hosts shielded when they moderate
Section 230 of the US Communications Act, added by the Telecommunications Act of 1996 on 8 February 1996, says that no provider or user of an "interactive computer service" shall be "treated as the publisher or speaker" of information that someone else provides, and protects "Good Samaritan" blocking of offensive material. The conference report on the bill says one of its purposes was to overrule Stratton Oakmont v. Prodigy and similar decisions, which had treated online services as the publishers of their users' posts because they had screened them. The section is still in force, amended in 1998 and 2018.
LessonMaking a host liable because it moderates gives it a reason not to moderate.
Credit
US Congress
2018
Hate outran the reviewers
In March 2018 UN investigators said Facebook had played a "determining role" in stirring up hatred against Rohingya Muslims in Myanmar, and Yanghee Lee, the UN special rapporteur on human rights in Myanmar, said it had "turned into a beast". In November 2018 Facebook published an assessment it had commissioned from BSR, a nonprofit, and wrote that the report concluded that before that year "we weren't doing enough to help prevent our platform from being used to foment division and incite offline violence". Facebook said it had by then hired 99 of the at least 100 native Burmese speakers it planned to have reviewing content by the end of 2018.
LessonA platform's rules are enforced only where it has reviewers and tools for the language people post in.
Credit
UN Independent International Fact-Finding Mission on Myanmar; BSR
2019
Copies blocked at upload
Hash matching compares a digital fingerprint, called a hash, of each upload with the hashes of material already identified. Microsoft and Dartmouth College built PhotoDNA in 2009 to find known images of child exploitation this way, and Microsoft gives it free to qualified organizations. In March 2019 the terrorist attack in Christchurch, New Zealand, was broadcast on Facebook Live, and Facebook hashed the video. By Facebook's count the live broadcast had fewer than 200 views and no user reports, and the first report came 29 minutes after the video began; in the next 24 hours Facebook removed about 1.5 million copies, more than 1.2 million of them blocked at upload, and shared more than 800 visually distinct versions with other companies through the shared database of the Global Internet Forum to Counter Terrorism (GIFCT).
LessonFingerprints stop copies only after someone has found the original.
Credit
Microsoft and Dartmouth College (PhotoDNA); Facebook; GIFCT members
2019
A hidden form, fined
Germany's Network Enforcement Act (NetzDG), in force from 1 October 2017, requires social networks with 2 million or more users in Germany to offer an easy-to-find procedure for complaints about unlawful content and to report on their handling every half-year. In July 2019 the Federal Office of Justice fined Facebook €2 million, the first fine under the law. Facebook's first report counted 1,704 complaints, only those sent through a NetzDG form placed on subpages of its legal notice, while YouTube reported about 215,000 and Twitter about 260,000.
LessonA count of complaints means little when the form for making them is hard to find.
Credit
Germany's Federal Office of Justice (Bundesamt für Justiz)
2021
Notes that need both sides
In January 2021 Twitter, now X, began Birdwatch, later renamed Community Notes, a pilot in which users write notes that add context to posts, and it published the code. A note is shown only when people from groups that usually rate differently find it helpful, a method its designers call bridging-based ranking. Isaac Slaughter, Axel Peytavin, Johan Ugander and Martin Saveski, of the University of Washington, Stanford and Yale, followed 40,078 posts with proposed notes from March to June 2023. They estimated that an attached note cut later reposts by 46.1% and likes by 44.1%, but reposts over a post's whole life by only 11.6%, and that notes that take about 48 hours to appear have almost no effect. In January 2025 Meta said it would end its third-party fact-checking program in the US and move to a Community Notes model.
LessonA correction helps only if it arrives before most of the sharing.
Credit
Twitter (now X); Slaughter, Peytavin, Ugander and Saveski (PNAS, 2025)
2021
Outside appeals that bind
The Oversight Board, an independent body of experts that hears appeals against Meta's decisions, published its first five decisions on 28 January 2021. It overturned four of Facebook's removals, upheld one and made nine policy recommendations. Its decisions on individual posts bind the company, which had seven days to restore the posts and 30 days to answer the recommendations in public. By the Board's count, more than 150,000 cases had been appealed to it since it began accepting them in October 2020.
Credit
Oversight Board
2023
Every EU decision logged
Since September 2023 the EU's Digital Services Act has required large online platforms to send the European Commission a "statement of reasons", the explanation a user is owed, for each moderation decision they take in the EU, and the Commission publishes them in the DSA Transparency Database. Amaury Trujillo, Tiziano Fagni and Stefano Cresci analyzed all 353.12 million records that the eight largest social media platforms sent in its first 100 days and found that "a remarkable fraction" of the data was inconsistent, X's most of all. On 20 November 2026 the database's front page showed 4,765,219,427 statements from 396 platforms in the previous 180 days, 46% of them fully automated decisions.
LessonA common format helps only if every platform fills it in the same way.
Credit
European Commission; Trujillo, Fagni and Cresci (ACM CSCW 2025)
Watch for
Models judged above humans
Meta says it trains large language models (LLMs), AI models that read and write text, on its Community Standards to judge whether posts break them, and that "we are beginning to see LLMs operating beyond that of human performance for select policy areas". It also uses them to take posts out of review queues when it is highly confident they break no rule. It did not name the policy areas or publish error rates.
Achieved whenan outside party compares the models' and human reviewers' error rates on the same sample of decisions and publishes the result.
Claimed by
Meta, May 2025
Mistakes cut in half
Meta estimated that in December 2024 one or two of every 10 of its removals may have been mistakes. After the changes it announced in January 2025, it said it had cut enforcement mistakes in the US in half between the fourth quarter of 2024 and the end of the first quarter of 2025, while prevalence, its estimate of how often people see violating content, stayed "largely unchanged" for most problem areas. It reported two rises on Facebook: bullying and harassment from 0.06–0.07% to 0.07–0.08%, and violent and graphic content from 0.06–0.07% to about 0.09%.
Achieved whenMeta publishes how it counts mistakes and an independent auditor confirms the fall.
Claimed by
Meta, May 2025
Open
01
Every Board case, many models
A public scorecard that gives large language models every case the Oversight Board has decided, with the Community Standards text that applied, and records whether each model would remove or keep the post and which rule it cites. Each answer is scored against Meta's original decision and the Board's final one, by policy area and language. Cases the Board published after a model's training data was gathered are scored separately, because a model may have read the older decisions.
The briefCredit · unclaimed02
Moderation tests in Burmese
A set of functional tests for moderation models in Burmese, Amharic and Tigrinya. A functional test is a short made-up post built to probe one known weakness, such as implicit hate, slang, or counter-speech that quotes a hateful remark in order to criticize it. Paid adult native speakers write and label every case, so no real person's post is used.
The briefCredit · unclaimed
Open 01 · the brief
Every Board case, many models
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public scorecard that gives large language models every case the Oversight Board has decided, with the Community Standards text that applied, and records whether each model would remove or keep the post and which rule it cites. Each answer is scored against Meta's original decision and the Board's final one, by policy area and language. Cases the Board published after a model's training data was gathered are scored separately, because a model may have read the older decisions.
Why it is still open
The only test found is a pilot by Nicolas Suzor and Lucinda Nelson of Queensland University of Technology (AoIR 2024), which gave GPT-4 the Board's hate speech decisions and found it could approximate them "in almost all current cases". The Board's other policy areas, other models, and Meta's own claim that LLMs beat human reviewers in some areas have not been tested in public.
Built with what exists
The Board's published decision pages; Meta's public Community Standards; open-weight models such as OpenAI's gpt-oss-safeguard-20b, Meta's Llama Guard 4 and Mistral Large 4 (see the Note "A trillion open parameters"); hosted models through their APIs; a short Python script for scoring.
First step
Put the Board's decisions in one policy area into a table with Meta's original call and the Board's outcome, and run two open-weight models on them.
Done looks like
Every published decision run through at least five models, three times each, with agreement rates against Meta's original call and against the Board shown by policy area and language. Prompts, outputs and scores are published, and a second person reruns 20 cases picked at random and gets the same scores.
Open 02 · the brief
Moderation tests in Burmese
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A set of functional tests for moderation models in Burmese, Amharic and Tigrinya. A functional test is a short made-up post built to probe one known weakness, such as implicit hate, slang, or counter-speech that quotes a hateful remark in order to criticize it. Paid adult native speakers write and label every case, so no real person's post is used.
Why it is still open
Test suites of this kind, after the English HateCheck, exist for ten languages in Multilingual HateCheck and for Indonesian, Tagalog, Thai and Vietnamese in SEAHateCheck (March 2026), which found models weakest in Tagalog. Burmese appears in SEA-SafeguardBench (December 2025), but that set tests chatbot prompts and replies, not posts. Burmese hate speech sets on GitHub, such as myHateSpeech, hold collected posts rather than functional tests, and no suite was found for Amharic or Tigrinya.
Built with what exists
HateCheck's published test templates; native-speaker writers and annotators paid through a university or an annotation firm; open moderation models such as Llama Guard 4, ShieldGemma and gpt-oss-safeguard, and hosted moderation APIs; a large language model to draft variants for the writers to correct.
First step
Adapt 200 English HateCheck templates into Burmese with two native speakers and measure how often two open models flag them.
Done looks like
At least 2,000 test cases per language, each labeled by three native speakers, with each model's accuracy per function set beside its accuracy on the English HateCheck, released under an open license with a statement of how the data was made.