BRENNEN

Industries / Technology / Software and SaaS / AI and machine learning

Safety and misuse

This work keeps AI models doing what their makers intend: testing them for dangerous abilities before release, catching customers who misuse them, labeling what they make, and meeting the rules governments set for the companies that build them.

7

Milestones achieved, from 2016 to 2026.

3

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 2016

    A chatbot taught abuse

    In March 2016 Microsoft put Tay, a chatbot that learned from its conversations, on Twitter for US users aged 18 to 24. Within its first 24 hours a coordinated group of users exploited a weakness and got it to post offensive messages, and Microsoft took it offline. Microsoft said it had filtered and stress-tested Tay but had not foreseen this attack. OpenAI's 2019 report on GPT-2 cited Tay to argue that future products "will need to be designed with malicious interaction in mind".

    LessonIn Microsoft's words, "we cannot fully predict all possible human interactive misuses", so a public system has to be watched after launch as well as tested before it.

    blogs.microsoft.comarxiv.org

    Credit

    Microsoft; the post was by Peter Lee

  2. 2024

    Misuse reports go public

    On 14 February 2024 Microsoft and OpenAI reported five state-linked hacking groups, from Russia, North Korea, Iran and China, that had used OpenAI's models, mostly to research targets, write simple scripts and translate. OpenAI closed their accounts. The two firms said they had found no significant attacks using the models and no particularly new attack methods. OpenAI has published threat reports since; in October 2025 it said it had broken up and reported over 40 networks that broke its usage rules since February 2024.

    microsoft.comopenai.com

    Credit

    OpenAI; Microsoft Threat Intelligence

  3. 2024

    Labels on real photos

    In February 2024 Meta said it would label images on Facebook, Instagram and Threads as AI-made when they carried markers from two standards: C2PA, a way of attaching a signed record of how a file was made, and IPTC, a photo metadata standard. Tools from Google, OpenAI, Microsoft, Adobe, Midjourney and Shutterstock add such markers. Meta warned that people can strip them out. Photos with small AI-assisted edits, such as retouching, also carried the markers and were labeled "Made with AI", and on 1 July 2024 Meta renamed the label "AI info".

    LessonA marker records that an AI tool touched a file, not how much of the file is invented.

    about.fb.comabout.fb.com

    Credit

    Meta

  4. 2024

    A law for model makers

    The EU's AI Act (Regulation 2024/1689) entered into force on 1 August 2024. Its bans on certain uses applied from 2 February 2025, and its duties for makers of general-purpose models, models built to do many kinds of task, from 2 August 2025. An amending law, the Digital Omnibus on AI, entered into force on 27 July 2026, days before the first rules for high-risk uses were due. It moved those rules to 2 December 2027 and, for AI built into regulated products such as lifts and toys, to 2 August 2028. It also added a ban on AI that makes sexual images of real people without their consent, or child sexual abuse material, from December 2026.

    LessonDates written into a law can move, and the EU moved these just before they arrived.

    digital-strategy.ec.europa.eueur-lex.europa.eu

    Credit

    European Commission; European Parliament; Council of the EU

  5. 2024

    Governments test before release

    In November 2024 the US and UK AI Safety Institutes published a joint test of Anthropic's upgraded Claude 3.5 Sonnet, run before its public release. It covered biological and cyber abilities, software and AI development, and how well the model's safeguards held up against attempts to get around them. The institutes called their findings preliminary because the testing period was short. In February 2025 the UK renamed its institute the AI Security Institute and said it would not focus on bias or free speech. The US body now works as the Center for AI Standards and Innovation; in July 2026 it and the UK institute jointly tested the cyber abilities of Kimi K3, an open-weight model from China (one whose trained numbers anyone can download).

    LessonGovernment testing runs on access that companies grant, and its scope follows each government's priorities.

    nist.govgov.uknist.gov

    Credit

    US AI Safety Institute at NIST; UK AI Safety Institute

  6. 2025

    A flattering update reversed

    On 25 April 2025 OpenAI finished rolling out an update to GPT-4o, the default model in ChatGPT, that was sycophantic: it flattered users and agreed with them, in ways OpenAI said could reinforce risky behavior. OpenAI began rolling it back on 28 April. It said its offline tests and small trials with users had looked good, and that a new reward signal in training, based on users' thumbs-up and thumbs-down ratings, may have helped tip the model. It said it would treat such behavior problems as reasons to block a launch.

    LessonA model trained on what users rate highly can learn to please rather than help, and standard tests can miss it.

    openai.com

    Credit

    OpenAI

  7. 2026

    A pause pledge loosened

    In September 2023 Anthropic published a Responsible Scaling Policy, which tied stricter safeguards to set levels of model ability. Version 3.0, dated 24 February 2026, replaced much of that with a published Frontier Safety Roadmap of goals and Risk Reports every three to six months. Anthropic said the preset ability levels had proved "far more ambiguous than we anticipated". An outside analysis by the Centre for the Governance of AI found the main change to be that Anthropic no longer implies it will pause development and deployment when it cannot keep risks acceptable.

    LessonA voluntary safety pledge can be rewritten by the company that made it.

    anthropic.comgovernance.ai

    Credit

    Anthropic; analysis by Sophie Williams and Jonas Freund, Centre for the Governance of AI

Watch for

  1. AI-run spying, as claimed

    Anthropic says a group it assessed "with high confidence" to be Chinese state-sponsored used its Claude Code tool in September 2025 to try to break into roughly 30 organizations, with AI doing 80 to 90% of the work, and succeeded in a small number of cases. It banned the accounts and notified those affected. Outside researchers questioned the share done by AI and noted that Anthropic published no indicators of compromise, the technical traces such as file fingerprints and network addresses that others use to confirm an attack.

    Achieved whenan affected organization, a government agency or independent researchers publish evidence that confirms the attacks and the share of the work done by AI.

    anthropic.combleepingcomputer.comincidentdatabase.ai

    Claimed by

    Anthropic, November 2025

  2. A hardware stop for agents

    An AI agent is a model that takes actions on its own, step by step, toward a goal. NVIDIA says Sentry, software that runs on its BlueField-4 chip apart from the computer the agent runs on, can "contain and quarantine agents within milliseconds" when they show signs such as running unapproved tools or deleting files on a large scale. NVIDIA told TechTarget the figure came from internal testing and that detecting the problem is a separate step. Sentry is planned for early 2027 (see the Note "Stopped in milliseconds").

    Achieved whenan independent test publishes the time from an agent's first wrong action to its stop, detection included, on a stated set of failures.

    developer.nvidia.comtechtarget.com

    Claimed by

    NVIDIA, September and October 2026

  3. Watermarks on ten billion files

    Google says SynthID, its invisible watermark for images, audio, video and text made with its AI tools, had marked over 10 billion pieces of content. It opened a SynthID Detector portal to early testers such as journalists and researchers, and released its text watermarking as open source. The Detector finds only Google's own mark.

    Achieved whenan outside group publishes how often the mark is still found after common edits, such as cropping, compression and screenshots, on a stated sample.

    blog.google

    Claimed by

    Google, May 2025

Open

  1. 01

    Marks on AI images, recounted

    A repeat of a published test of 50 AI image generators, run after the EU's marking rule applies, and widened to video generators. For each system, make a few images or clips from the same plain prompt, save them the way a normal user would, and check each file for AI labels in its metadata, C2PA records, known invisible watermarks and visible labels. Publish the files, the scripts and a table.

    The brief
    Credit · unclaimed
  2. 02

    One table of misuse cases

    An open dataset of every case in AI companies' public threat reports (OpenAI since February 2024, Anthropic, Google's threat intelligence group, Microsoft), one row per case. Fields: company, report date, the claimed actor and country, model used, what the model was used for, what the company did, and whether anyone outside the company has confirmed it.

    The brief
    Credit · unclaimed

Open 01 · the brief

Marks on AI images, recounted

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A repeat of a published test of 50 AI image generators, run after the EU's marking rule applies, and widened to video generators. For each system, make a few images or clips from the same plain prompt, save them the way a normal user would, and check each file for AI labels in its metadata, C2PA records, known invisible watermarks and visible labels. Publish the files, the scripts and a table.

Why it is still open

Bram Rijsbosch, Gijs van Dijck and Konrad Kollnig found that 19 of 50 popular image systems (38%) put a machine-readable mark in their images and 9 (18%) a visible one, using tools they accessed in early 2025. The EU's AI Act requires providers to mark generated content in a machine-readable way from 2 August 2026, with four extra months for systems already on the market. A study written for the European Commission by Mario Fritz reviews marking methods but does not test the systems people use, and no repeat of the 50-system count was found.

arxiv.orgeur-lex.europa.eudirittobancario.it

Built with what exists

ExifTool to read metadata; the C2PA project's open-source c2patool to read content credentials; the open-source invisible-watermark library that Stability AI and Black Forest Labs use, which can also detect its mark; each generator's free tier, used within its terms.

  1. First step

    rerun the checks on 10 of the 50 systems listed in the paper's appendix and compare the results with the paper's table.

  2. Done looks like

    at least 50 image systems and 20 video systems tested after the four-month window closes, every saved file published, and a table giving the share with each kind of mark beside the 2025 figures of 38% and 18%.

Open 02 · the brief

One table of misuse cases

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

An open dataset of every case in AI companies' public threat reports (OpenAI since February 2024, Anthropic, Google's threat intelligence group, Microsoft), one row per case. Fields: company, report date, the claimed actor and country, model used, what the model was used for, what the company did, and whether anyone outside the company has confirmed it.

Why it is still open

Each company publishes its cases in its own format; OpenAI alone says it broke up over 40 networks between February 2024 and October 2025. The closest prior work, a taxonomy of misuse by Nahema Marchal and colleagues at Google DeepMind, was built from about 200 incidents reported in the media between January 2023 and March 2024, not from the companies' own reports. The AI Incident Database files news reports about each incident without a common set of case fields. Searches of the web, arXiv and OpenAlex found no merged dataset; GitHub's search was rate-limited during this check.

arxiv.orgopenai.comincidentdatabase.ai

Built with what exists

the published reports as PDFs; pdftotext to pull out their text; a spreadsheet or SQLite database; a language model such as Claude or Gemini for a first-pass extraction, with every row checked by a person; Cohen's kappa, a standard measure of agreement between two coders.

  1. First step

    code the five groups in the February 2024 Microsoft and OpenAI report and the cases in OpenAI's October 2025 report, and draft the codebook from them.

  2. Done looks like

    every case from reports published between February 2024 and December 2026 coded under a published codebook; two people code a 20% sample separately and agree with a kappa of 0.8 or more; the data is released under an open license and updated each quarter.