Industries / Technology / AR and VR
Tracking and input
Tracking is how a headset measures where the head, hands and eyes are, many times a second, so the picture moves as the wearer does; input is how the wearer gives commands.
7
Milestones achieved, from 1968 to 2025.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1968
An arm from the ceiling
Sutherland's display measured the head with "a mechanical arm hanging from the ceiling", each joint read by a digital shaft encoder. His report calls it "rather heavy and uncomfortable to use" and says "We built it to have a sure method of measuring head position." An ultrasonic alternative, with three transmitters on the head and four receivers in the ceiling, built by Charles Seitz and Stylianos Pezaris at MIT Lincoln Laboratory, worked "for a few minutes before cumulative errors were objectionable".
Lessonthe sensor that could not drift was too heavy to wear comfortably, and the light one drifted.
Credit
Ivan Sutherland, Harvard; Charles Seitz and Stylianos Pezaris
1979
Magnetic tracking
Frederick Raab, Ernest Blood, Terry Steiner and Herbert Jones of Polhemus Navigation Sciences described a tracker in which magnetic fields from a small source give "both the position and orientation of the sensor relative to the source", with no need for a clear line of sight, and discussed its use for tracking a pilot's head. The paper says magnetic trackers already served in helmet-mounted sights, and that nearby metal distorts the fields and causes "significant measurement errors".
Lessona tracker that needs no line of sight can be thrown off by the metal around it.
Credit
Raab, Blood, Steiner and Jones, Polhemus
2016
Lasers sweep the room
Valve's Lighthouse base stations give a flash and then sweep a laser line across the room. Sensors on the headset time the sweep, and, in Valve's words, "Time is equivalent to angle". Alan Yates of Valve says he invented it. It reached buyers with the HTC Vive, which began shipping on 5 April 2016 with two base stations.
Credit
Alan Yates, Valve; HTC
2019
Cameras on the headset
"Previously, VR devices relied on external sensors to track these movements," Meta wrote of Oculus Insight, which tracks the Rift S and Quest with the headset's own cameras by mapping the room as it goes (SLAM, simultaneous localization and mapping). Meta calls it the first full-featured inside-out system for consumer VR, while Road to VR calls the Windows Mixed Reality headsets of 2017 "some of the first headsets with inside-out tracking". From December 2019 the Quest's monochrome cameras also tracked bare hands. Abdlkarim and colleagues later measured Quest 2 hand tracking against twelve motion-capture cameras: fingertip position was off by 1.1 cm on average, joint angles by 9.6 degrees, and the delay was 45.0 ms.
tech.facebook.comroadtovr.comdevelopers.meta.comlink.springer.com
Credit
Oculus (Facebook, now Meta), with Oskar Linde, Joel Hesch, Anna Kozminski and David Vogt named by Meta
2024
Looking as pointing
The HTC Vive Pro Eye, with eye tracking by Tobii, went on sale in June 2019 at US$1,599. PlayStation VR2, from 22 February 2023, used eye tracking to draw full detail only where the wearer looks, which is called foveated rendering. Apple Vision Pro, on sale from 2 February 2024, made the eyes the main pointer: the wearer looks at an item and taps two fingers together to select it.
Credit
Tobii and HTC; Sony Interactive Entertainment; Apple
2024
Headsets left behind
Microsoft deprecated Windows Mixed Reality and removed it from Windows 11 version 24H2. Existing headsets keep working with Steam only through November 2026, and only on version 23H2. Road to VR wrote that the camera-tracked headsets from Acer, Asus, Dell, Lenovo, HP and Samsung had "essentially been bricked".
Lessoncamera tracking runs in the platform owner's software, so working hardware stops when the platform goes.
Credit
Microsoft
2025
Signals from the wrist
In Nature in July 2025, Patrick Kaifosh, Thomas Reardon and a team at Reality Labs described a wristband that reads the electrical activity of the forearm muscles (surface electromyography, EMG) and decodes it with models "that generalize across people", so a new user needs no calibration. In tests, it made 0.88 gesture detections a second, and people wrote by hand at 20.9 words a minute. Meta began selling the band, the Meta Neural Band, with its Ray-Ban Display glasses in September 2025.
Credit
Patrick Kaifosh, Thomas Reardon and the CTRL-labs team at Reality Labs, Meta
Watch for
Detail only where you look
Valve says Steam Frame's foveated streaming, which uses eye tracking to send the sharpest part of a picture streamed from a PC only where the wearer is looking, "typically offers over a 10x improvement in image quality and effective bandwidth". The headset went on sale in September 2026. Gamers Nexus measured its streaming delay at 46.1 ms against 72.5 ms for Quest 3 over Wi-Fi, but wrote that "there's not much we can do to demonstrate it in video recordings".
Achieved whenan outside lab publishes image quality at matched bandwidth with the feature on and off, and the delay from an eye movement to the sharpened picture.
Claimed by
Valve, November 2025
Writing with a finger
Meta said in September 2025 that Neural Band users would "in the near future, even write out messages" by tracing letters. Handwriting entered early access in the US in January 2026, in English, for WhatsApp and Messenger, and was still in early access in August 2026. Neither report gives a speed or accuracy, and the Nature paper's 20.9 words a minute was a laboratory figure.
Achieved whenthe feature is released to all users and an outside test measures words per minute and errors on bands bought in shops.
Claimed by
Meta, September 2025 and January 2026
Open
01
Hand tracking across makers
A rig in which a robot arm moves a lifelike model hand along the same paths in front of each headset, while software logs where the headset places each finger joint and compares it with the arm's known positions.
The briefCredit · unclaimed02
Gaze selection across makers
A test of how small a target each headset lets people pick by looking and pinching, and how fast, run with the same volunteers and targets on Vision Pro, Galaxy XR, Steam Frame and a Meta headset with eye tracking.
The briefCredit · unclaimed
Open 01 · the brief
Hand tracking across makers
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A rig in which a robot arm moves a lifelike model hand along the same paths in front of each headset, while software logs where the headset places each finger joint and compares it with the arm's known positions.
Why it is still open
Published tests cover one maker at a time. Abdlkarim and colleagues measured only the Quest 2 and offered "a repeatable methodology to measure the hand-tracking accuracy of other devices". Godden, Steedman and Pan moved a test hand with a KUKA robot arm, but tested "two recent mixed reality devices from Meta" (IEEE TVCG, 2025). In 2025 Hu and colleagues at Duke University compared five makers' headsets, Apple Vision Pro among them, for head tracking only. A scan of the papers citing the first two studies found no comparison of hand tracking across makers.
Built with what exists
a robot arm such as the UFACTORY xArm, which its maker rates to repeat positions within 0.1 mm or better; a silicone model hand; logging apps written against OpenXR's hand-tracking extension, XR_EXT_hand_tracking, for Meta, Android XR and SteamVR headsets, and against Apple's HandTrackingProvider for Vision Pro.
First step
Write the OpenXR logging app, fix the model hand to the arm, and record one straight path on a Quest 3.
Done looks like
At least five headsets from at least three makers, the same 20 paths repeated 10 times each in two light levels, with fingertip error in millimetres, joint-angle error in degrees and delay in milliseconds published per headset, along with raw logs and code.
Open 02 · the brief
Gaze selection across makers
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A test of how small a target each headset lets people pick by looking and pinching, and how fast, run with the same volunteers and targets on Vision Pro, Galaxy XR, Steam Frame and a Meta headset with eye tracking.
Why it is still open
Studies comparing eye-tracking accuracy across makers cover older headsets. In October 2026 Aleksandr Chernysh, Oleg Komogortsev and colleagues at Texas State University measured median errors of 1.04° on the Meta Quest Pro, 1.50° on the HTC Vive Pro Eye and 2.41° on the HTC Vive Focus Vision. Vision Pro cannot be tested the same way, because "visionOS does not share eye input with apps or websites, or even Apple", Apple says; apps learn only what the wearer selects. A test built on selections works on every headset.
Built with what exists
a grid of targets of set angular sizes, from 0.5 to 4 degrees, built in Unity with OpenXR for Android XR and SteamVR headsets and in SwiftUI for visionOS, logging hits, misses and times; OpenXR's eye-gaze extension, XR_EXT_eye_gaze_interaction, to log raw gaze where a platform offers it, as a second measure; a chin rest to hold the head in place.
First step
Build the target grid for one headset and run it with five volunteers.
Done looks like
Hit rate and time by target size and position for at least four headsets, with at least 20 volunteers on each, the smallest target that 95% of selections hit, and data and code published.