BRENNEN

Industries / Technology / AR and VR

Work and training

Headsets and glasses are used at work to train staff, to guide assembly and repair step by step, and to check that a job was done right.

6

Milestones achieved, from 1992 to 2026.

1

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1992

    A name for overlays

    Thomas Caudell and David Mizell of Boeing Computer Services described a headset that would show wiring instructions over "the wiring formboard, where wire bundles are manufactured by laying individual wires across pegs", and wrote: "we refer to the technology as 'augmented reality' (AR)". Their test rig used a Private Eye head-mounted display. A Boeing technical fellow wrote in 2018 that the two "coined the term 'augmented reality'".

    mona.uwi.edusme.org

    Credit

    Thomas Caudell and David Mizell, Boeing

  2. 2016

    Glasses on the harness line

    In a Boeing pilot, technicians wiring harnesses followed instructions on Google Glass running Upskill's Skylight software. A Boeing engineer told CIO that it would "decrease the assembly time by 25 percent and it significantly reduced error rates", on "an isolated internal test network", not the live system. Upskill's account says errors fell to zero.

    Lessonearly results came from pilots, and the user's and the vendor's figures differed.

    cio.comarinsider.co

    Credit

    Boeing; Upskill

  3. 2018

    Headsets in every store

    Walmart said it would send "four headsets to every Walmart supercenter and two units to every Neighborhood Market", "more than 17,000 Oculus Go headsets in stores by the end of the year", to train staff with software from STRIVR. Walmart said the training improved test scores by 10 to 15 percent; no outside measurement was found.

    corporate.walmart.com

    Credit

    Walmart; STRIVR

  4. 2020

    Fasteners placed in minutes

    Lockheed Martin technicians building NASA's Orion spacecraft for Artemis II used HoloLens 2 headsets to place fasteners called click bonds. Lockheed reported "90% less time", with work that took 8 hours done in 45 minutes, and NASA's Carlos Garcia said the method saved up to three weeks.

    nasa.gov

    Credit

    Lockheed Martin, with Scope AR's WorkLink software; NASA

  5. 2020

    Equal, not better

    Kaplan and colleagues pooled studies of training in virtual, augmented and mixed reality (Human Factors, online February 2020, in print 2021). They found that such training "does not express a different outcome than training in a nonsimulated, control environment. It is equally effective", and placed its value where "danger or cost may make traditional training impossible".

    Lessonon average a headset course matches ordinary training but does not beat it.

    api.crossref.org

    Credit

    Kaplan and colleagues

  6. 2026

    Instructions lose their app

    Microsoft confirmed in October 2024 that HoloLens 2 production had ended. Dynamics 365 Guides, its app for writing step-by-step instructions shown in the headset, and its Remote Assist app could be bought or renewed only until 1 November 2025, and they reach end of support on 31 December 2026, after which "The apps will no longer be functional".

    Lessoninstructions written in one vendor's app end when that app does.

    uploadvr.comlearn.microsoft.com

    Credit

    Microsoft

Watch for

  1. Instructions written by AI

    Augmentir says its generative AI can "transform existing content—such as training videos, existing SOPs" (standard operating procedures) "into native step-by-step, interactive procedures", now with AR features, and reports an "82% reduction in new hire onboarding time" across its customers. The release does not say that steps are pinned to real parts, and the figure is the company's own.

    Achieved whenan outside test compares AI-converted instructions with the originals on a real task, counting errors and time.

    augmentir.com

    Claimed by

    Augmentir, September 2025

Open

  1. 01

    Catch the wrong step live

    A system that watches an assembly through a headset's cameras and warns the worker the moment a step is done wrong or skipped, tested live on a real product with a published tally of mistakes caught, mistakes missed and false alarms.

    The brief
    Credit · unclaimed
  2. 02

    Instructions from the manual

    A tool that reads an existing assembly manual, a PDF with drawings, and turns it into steps shown in a headset, each pinned to the real part it concerns, followed by a test with first-time users against the paper manual.

    The brief
    Credit · unclaimed

Open 01 · the brief

Catch the wrong step live

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A system that watches an assembly through a headset's cameras and warns the worker the moment a step is done wrong or skipped, tested live on a real product with a published tally of mistakes caught, mistakes missed and false alarms.

Why it is still open

Research tests run on recorded video. In HoloAssist (2023), a set of recordings made on HoloLens 2 by Microsoft and others, mistakes were "around 6%" of actions and the best mistake detector scored an F-score, a measure that combines hits and false alarms, of 40.19 out of 100. The IndustReal method (2024) runs at "178 fps", fast enough for live use, but was tested offline and "fails to generalize to (out-of-distribution) execution errors", the kinds it was not trained on. The authors of PREGO (2024) wrote that "no technique can currently detect open-set procedural mistakes online".

arxiv.orgarxiv.orgarxiv.org

Built with what exists

a headset whose cameras apps can read, such as HoloLens 2 with the open hl2ss streaming library; IndustReal's released data and 3D part models; a model that answers fixed questions about each frame, such as whether the red wire is on peg 4, for example Gemini Decision 1 (see the Note "A thousand questions, one pass"); a worktable with a fixed kit, such as IndustReal's toy car.

github.com

  1. First step

    Run IndustReal's released model on live video of ten assemblies of its toy car and log what it catches.

  2. Done looks like

    At least 200 live assemblies by 20 people, with errors planted on a schedule the system does not know, scored against a human observer, and the catch rate, miss rate and false alarms per hour published with video and labels.

Open 02 · the brief

Instructions from the manual

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A tool that reads an existing assembly manual, a PDF with drawings, and turns it into steps shown in a headset, each pinned to the real part it concerns, followed by a test with first-time users against the paper manual.

Why it is still open

Research systems stop short of this. PaperToPlace (UIST 2023) turned a photo of a paper instruction sheet into steps placed in mixed reality on a Meta Quest Pro, but could pin them only to fixed objects, citing a "lack of support for real-time arbitrary object tracking with Quest Pro", and was tested on kitchen tasks. Satori (CHI 2025) guided 16 people through everyday tasks with GPT-4V on HoloLens. Microsoft previewed Copilot for Dynamics 365 Guides on HoloLens 2 in November 2023, and Guides itself ends in December 2026.

arxiv.orgarxiv.orgcomputerworld.com

Built with what exists

Vuforia Model Targets, which track a real object by its shape from a CAD model or 3D scan; object tracking in Apple's visionOS, which learns reference objects in Create ML; a model that reads long documents with drawings, such as Claude Fable 5.2 (see the Note "Three working devices"); Satori's open code.

developer.vuforia.comdeveloper.apple.comgithub.com

  1. First step

    Take one public PDF manual for a flat-pack product, have a model write the step list with part names and drawing regions, and check it by hand.

  2. Done looks like

    Manuals for three real products converted with every hand correction logged, and 30 first-time users per group, headset or paper, timed and scored for errors on each product and retested a week later, with results and code published.