BRENNEN

Industries / Technology / Software and SaaS / Cybersecurity

Training people

Teaching staff to spot and report tricks such as phishing, emails that pose as a trusted sender to get someone to click a link or give up a password, and giving defenders practice at running security under attack.

7

Milestones achieved, from 2004 to 2026.

1

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 2004

    A fake colonel tests cadets

    On 22 June 2004 the US Military Academy at West Point emailed 512 randomly chosen cadets a message from a made-up colonel, Robert Melville, asking them to click a link about a problem with their grades. More than 400 of them, 80%, clicked, and so did 90% of first-year cadets who had just had four hours of computer security instruction. The academy's chief information officer refused to run it as a "gotcha" with scolding emails, saying that would undermine the trust between the cadets and his office, so the clicks were used to show the size of the problem instead.

    LessonClassroom instruction did not stop people from clicking a lure that arrived at the right moment.

    er.educause.edu

    Credit

    Aaron J. Ferguson, an NSA visiting professor at West Point, and the academy's cadet information security officers

  2. 2005

    Students defend a live network

    The Center for Infrastructure Assurance and Security at the University of Texas at San Antonio hosted the first Collegiate Cyber Defense Competition in April 2005, for five schools and 40 students. Each team is given an operational business network that it must secure, operate and defend. Teams score for detecting and answering outside attacks, keeping services such as mail and web servers running, and handling business requests. By the center's count, 241 schools and 2,880 students took part in 2019.

    cias.utsa.educisse.info

    Credit

    Gregory B. White and Dwayne Williams, Center for Infrastructure Assurance and Security, UTSA

  3. 2009

    A lesson after the click

    Carnegie Mellon University researchers built PhishGuru, which sends staff simulated phishing emails and shows a short lesson to anyone who clicks, at what they called a "teachable moment". In a 28-day study of 515 university staff and students, among people who had fallen for the first email, 14.7% of those trained twice gave information to a fake site on day 28, against 31.4% of those trained once. This "embedded training" is now one of the two standard forms of workplace security training, alongside an annual course, as an outside 2025 study describes it.

    cups.cs.cmu.edusysnet.ucsd.edu

    Credit

    Ponnurangam Kumaraguru, Lorrie Faith Cranor, Jason Hong, Alessandro Acquisti and colleagues, Carnegie Mellon University

  4. 2020

    A fake bonus backfires

    In December 2020 GoDaddy, a web-hosting company, emailed staff the offer of a $650 holiday bonus. It was a phishing test, and two days later about 500 employees who had clicked were told they had failed it and that no bonus was coming. After staff complaints and press reports, a spokesperson apologized and said "we need to do better and be more sensitive to our employees".

    LessonA test built on staff's real hopes can turn them against the security team it is meant to help.

    engadget.com

    Credit

    reported first by Lorraine Longhi of the Copper Courier

  5. 2022

    Training made some worse

    Over 15 months, researchers at ETH Zurich sent simulated phishing emails to more than 14,000 employees of a partner company. They found that embedded training did not make employees more resilient and could make them more likely to fall for later tests. The same study found that employees reporting suspicious emails worked as a shared alarm for the whole organization: reports picked up new phishing campaigns quickly, at a workload the organization could handle.

    LessonA staff member's report can protect everyone, while a lesson after a click may protect nobody.

    arxiv.org

    Credit

    Daniele Lain, Kari Kostiainen and Srdjan Čapkun, ETH Zurich, in a paper for the IEEE Symposium on Security and Privacy 2022

  6. 2025

    Annual training, no measurable effect

    An eight-month randomized trial at UC San Diego Health sent ten simulated phishing campaigns to more than 19,500 employees. Whether someone had recently finished the annual awareness course made no significant difference to whether they failed a test. Embedded training lowered failure rates by 1.7 percentage points on average, and over half of training sessions ended within 10 seconds. The choice of lure mattered far more: a dull one drew clicks from 1% to 2% of staff, others over 30%. A second trial, of 12,511 staff at a US financial technology firm, found no significant effect of training on clicks or on reports.

    LessonA training result means little without a control group that got no training and a measure of how hard the test email was.

    sysnet.ucsd.eduarxiv.org

    Credit

    Grant Ho, Ariana Mirian, Stefan Savage, Geoffrey Voelker and colleagues, UC San Diego; Andrew Rozema and James Davis, Purdue University

  7. 2026

    AI writes lures like experts

    In a study with 101 participants, spear-phishing emails (phishing aimed at one named person) that an AI tool researched and wrote with no human help drew clicks from 54% of recipients. Emails written by human experts also drew 54%, and a generic control email 12%. The tool built accurate profiles of its targets in 88% of cases. The study was published in Expert Systems with Applications in June 2026, after a first version in November 2024.

    LessonSpelling mistakes and generic greetings, the signs much training teaches, are no longer reliable signs of phishing.

    schneier.comarxiv.org

    Credit

    Fred Heiding, Bruce Schneier, Arun Vishwanath and colleagues

Watch for

  1. Big drops, no control group

    KnowBe4, which sells security awareness training, says that across 42 million simulated phishing emails sent to 14.8 million users at 64,000 organizations, the share of staff who fall for a test drops from 33.2% before training to 20.1% after 90 days and 4.2% after a year. It reports this as a 79% fall. The release does not mention an untrained comparison group, and the randomized trials above found effects of a few percentage points at most.

    Achieved whenan outside team finds a fall of that size in a randomized trial with an untrained control group and test emails of matched difficulty.

    knowbe4.com

    Claimed by

    KnowBe4, July 2026

Open

  1. 01

    A rerunnable training trial

    An open kit for running a randomized phishing-training trial inside one organization that has approved it in writing: a protocol and pre-registration template, random assignment of staff to trained and untrained groups, test emails rated for difficulty, and analysis code that reports clicks and reports with confidence intervals. The kit sends only simulated emails to the organization's own staff.

    The brief
    Credit · unclaimed
  2. 02

    Lure difficulty against clicks

    A public table of every simulated phishing email described in published studies, each with its reported click rate, audience and date, and a difficulty rating under the NIST Phish Scale from two human raters and from a language model. Then a published test of how much of the spread in click rates the rating explains.

    The brief
    Credit · unclaimed

Open 01 · the brief

A rerunnable training trial

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

An open kit for running a randomized phishing-training trial inside one organization that has approved it in writing: a protocol and pre-registration template, random assignment of staff to trained and untrained groups, test emails rated for difficulty, and analysis code that reports clicks and reports with confidence intervals. The kit sends only simulated emails to the organization's own staff.

Why it is still open

The two largest randomized trials found little or no effect from training, while the largest vendor reports big drops from customer data with no control group. Neither trial published code or data with its paper; the UC San Diego team writes that the terms of its health system's contract with Proofpoint, the training vendor, kept it from showing any of the test emails. A search of GitHub in November 2026 found open-source phishing simulators, such as GoPhish and CESNET's Phishingator, and hundreds of test-email templates, but no kit for a controlled trial.

sysnet.ucsd.eduarxiv.orggithub.com

Built with what exists

GoPhish or Phishingator, open-source tools for sending simulated phishing; the NIST Phish Scale User Guide (NIST Technical Note 2276) to rate difficulty; Python with pandas and statsmodels; OSF, the Open Science Framework, for pre-registration; an AI coding agent such as Claude Code to write the assignment and analysis code (see the Note "Effort, not model").

  1. First step

    Write the pre-registration and sample-size calculation from the published UC San Diego design and post it on OSF for comment.

  2. Done looks like

    Two organizations other than the builder's, each with at least 1,000 staff, run the kit for six months under their own written approval and publish pre-registered results against an untrained control group, using the same analysis code.

Open 02 · the brief

Lure difficulty against clicks

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public table of every simulated phishing email described in published studies, each with its reported click rate, audience and date, and a difficulty rating under the NIST Phish Scale from two human raters and from a language model. Then a published test of how much of the spread in click rates the rating explains.

Why it is still open

NIST built the Phish Scale so that click rates could be read against how hard each email was to spot, and its authors wrote that it still needed validating "with a larger variety of phishing emails". The UC San Diego trial found the choice of email moved click rates from 1% to over 30%, and Purdue's trial rated its own emails with the scale (7.0% clicks on easy ones, 15.0% on hard ones) without publishing them. A team publishing as UdeC-LLM-Security has rewritten the scale's questions for language models, in a paper for the ICAART 2026 conference, but released the questions, not a rated set of emails.

csrc.nist.govarxiv.orggithub.com

Built with what exists

the NIST Phish Scale User Guide; lure descriptions and click tables from published papers; a language model such as Claude given the UdeC question set; Python, with Cohen's kappa to measure how often raters agree.

  1. First step

    Copy the test emails and click rates from three papers (West Point 2004, Carnegie Mellon 2009, UC San Diego 2025) into a spreadsheet and rate each one.

  2. Done looks like

    At least 100 test emails from at least eight published studies, each linked to its source page; agreement between the two human raters and between humans and the model reported; the fit of click rate against difficulty published with the data.