Industries / Technology / Software and SaaS / Cloud and data centers / Edge computing
Running AI at the edge
This is the work of running trained AI models that see, hear or handle language on small computers on site, in a camera, a shop or a vehicle, instead of sending the data to a distant data center. It covers the accelerator chips and boards, the software that shrinks and runs the models, and the shared tests that measure them.
7
Milestones achieved, from 2014 to 2026.
3
To watch for: announced, not yet confirmed outside.
3
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
2014
Graphics chips for small devices
On 25 March 2014 NVIDIA announced the Jetson TK1, a small board built on its Tegra K1 chip, with 192 graphics cores that developers program in CUDA, NVIDIA's language for doing many sums at once. NVIDIA rated it at 326 gigaflops (billions of operations a second) and called it "the world's first mobile supercomputer for embedded systems", its own claim; BDTI, an outside analysis firm, reported preorders at $192.
Credit
NVIDIA
2017
A common runtime for devices
On 14 November 2017 Google released a developer preview of TensorFlow Lite, a runtime (the program that runs a trained model) for phones and embedded devices, with a core of about 70 KB. On 4 September 2024 Google renamed it LiteRT; by Google's count it then ran in more than 100,000 apps on 2.7 billion devices.
Credit
Google's TensorFlow team
2017
AI on a USB stick
Movidius showed its Fathom stick in April 2016; after Intel bought the company, it went on sale on 20 July 2017 at $79 as the Movidius Neural Compute Stick. It holds a Myriad 2 vision chip and lets a small computer such as a Raspberry Pi run a trained network offline. Intel stopped shipping the second-generation stick on 30 June 2022 and ended technical support on 30 June 2023.
LessonA plug-in accelerator lasts only as long as its maker supports it; Intel sent stick users to a different product.
Credit
Movidius; Intel
2019
A shared test for inference
On 6 November 2019 the MLPerf group published its first inference results, inference meaning running a trained model: 595 results from 14 organizations, on systems ranging from "embedded devices and smartphones to large-scale data center systems". NVIDIA described two of its four test patterns, single-stream (one input at a time, as from one camera) and multi-stream (several inputs at a fixed rate), as the edge-focused ones. On 16 June 2021 MLCommons added MLPerf Tiny, for models of about 100 KB on very low-power chips.
Credit
the MLPerf group, now MLCommons (paper by Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson and others)
2023
A cloud-tied kit withdrawn
Microsoft announced Azure Percept on 2 March 2021, a kit with a camera module built on Intel's Myriad X chip, tied to Azure cloud tools for building models and sending them to the device. Microsoft retired the kit, its audio accessory and the cloud services behind them on 30 March 2023, saying they had been in preview.
LessonWhen the maker withdraws the cloud side, the device loses its updates and support, and users' cloud projects that were not closed kept billing.
Credit
Microsoft
2025
Robot computer runs language models
On 25 August 2025 NVIDIA began selling the Jetson AGX Thor developer kit at $3,499, a computer for robots with 128 GB of memory that works within 130 watts. The same day the review site ServeTheHome measured 149.1 tokens (word pieces) per second on the Llama 3.1 8B language model, against the 150.8 NVIDIA expected.
Credit
NVIDIA; measured by ServeTheHome
2026
Accelerator slower than its host
Raspberry Pi has sold a $70 AI Kit with a Hailo-8L chip, rated at 13 TOPS (trillions of operations a second), since 4 June 2024. On 15 January 2026 it released the $130 AI HAT+ 2, with a Hailo-10H chip rated at 40 TOPS on 4-bit numbers and 8 GB of its own memory, sold for running small language models. In tests published that day, Jeff Geerling found the Raspberry Pi 5's own processor faster than the Hailo-10H on most models, with the Hailo close only on Qwen2.5 Coder 1.5B, though the Hailo was slightly more efficient in most cases.
LessonA TOPS rating does not predict language-model speed; the Hailo is held to 3 W, while the Pi's processor may draw 10 W.
Credit
Raspberry Pi and Hailo; tested by Jeff Geerling
Watch for
Open NPU design and compiler
An NPU (neural processing unit) is a processor built to run AI models. Google says its open Coral NPU design now has a matrix unit doing 256 multiply-accumulates per clock tick, "configurable for up to 1 TOPS and targeting 10mW average power for inference", and that its open Coral Compiler v1.0 takes JAX, PyTorch and TensorFlow models to the hardware. Google's own documents show the compiler's output running in simulation, while the Synaptics Coralboard SL that went on sale the same day uses Synaptics' own Torq compiler (see the Note "An open compiler").
Achieved whena chip containing the new matrix unit is on sale and someone outside Google and Synaptics runs the compiler's output on it and measures power.
Claimed by
Google Research, October 2026
Small models that run faster
Liquid AI says its LFM3-1.7B text model produces text 1.8 to 2.8 times faster than the next-fastest comparable model and uses the least memory, in a test with llama.cpp and 4-bit weights on three devices it does not name. It also says each of its six LFM3 models can be fine-tuned, trained further for one job, on a single GPU (see the Note "One GPU to fine-tune").
Achieved whensomeone outside Liquid AI repeats the speed comparison on named hardware, such as a Raspberry Pi 5, with published settings, and gets a similar ratio.
Claimed by
Liquid AI, October 2026
Agent tasks 6.4 times faster
NVIDIA says its TensorRT Edge-LLM software finished MLPerf's new edge agentic test, 1,007 turns of tool-calling work with the Qwen3.6-27B model, on one Jetson AGX Thor in 24 minutes 36 seconds, 6.4 times faster than a llama.cpp reference run of 2 hours 37 minutes, with 87.94% function-calling accuracy. The comparison is NVIDIA's own, the two runs stored the model's numbers in different formats (NVFP4 against Q4_K_M), and no power figure was given.
Achieved whensomeone outside NVIDIA runs both setups on one Thor under MLPerf's accuracy check and gets a similar ratio.
Claimed by
NVIDIA, September 2026
Open
01
Energy per answer, cheap boards
A public table of energy per correct answer for two or three small open language models, such as Qwen2.5 1.5B and Llama 3.2 1B, on low-cost boards: a Raspberry Pi 5 alone, a Raspberry Pi 5 with the AI HAT+ 2, and a Jetson Orin Nano Super. Power is measured at the wall while each board answers a fixed set of questions with known answers. What makes it possible: open models of 1 to 2 billion parameters now run on boards costing $100 to $250, and ggml 1.0.0, the library under llama.cpp, gives a fixed software version to test against (see the Note "A first stable number for local AI").
The briefCredit · unclaimed02
An outside Coralboard test
Compile the four MLPerf Tiny reference models, which detect keywords, detect whether a person is in view, sort small images into classes and spot abnormal machine sounds, with Synaptics' open Torq compiler, and run them on the Coralboard SL's NPU. For each, report accuracy against the same model run in LiteRT on a PC, and milliseconds and millijoules per inference. What makes it possible: Google's open Coral NPU design, and Synaptics' open compiler and board, on sale since 20 October 2026.
The briefCredit · unclaimed03
Blade defects on small boards
Liquid AI published a worked example in which its LFM3-VL-2B model is fine-tuned on WTBs2025, a set of wind-turbine blade photos (CC BY 4.0). Repeat that fine-tune, then apply quantization, storing the model's numbers with fewer bits, by converting it to 4-bit GGUF files; run it with llama.cpp on a Raspberry Pi 5 and a Jetson Orin Nano Super and score it on the same 1,372 held-out photos. The work uses photos only and never touches a working turbine. What makes it possible: small open vision models that can be fine-tuned on one GPU (LFM3, October 2026).
The briefCredit · unclaimed
Open 01 · the brief
Energy per answer, cheap boards
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public table of energy per correct answer for two or three small open language models, such as Qwen2.5 1.5B and Llama 3.2 1B, on low-cost boards: a Raspberry Pi 5 alone, a Raspberry Pi 5 with the AI HAT+ 2, and a Jetson Orin Nano Super. Power is measured at the wall while each board answers a fixed set of questions with known answers. What makes it possible: open models of 1 to 2 billion parameters now run on boards costing $100 to $250, and ggml 1.0.0, the library under llama.cpp, gives a fixed software version to test against (see the Note "A first stable number for local AI").
Why it is still open
MLPerf Inference v6.1, published on 16 September 2026, had 24 edge results from five organizations; the only board-sized system was the $3,499 Jetson AGX Thor kit, the rest desktops and servers, and the round's summary file marks none of its 492 results as carrying a power measurement. Jeff Geerling's January 2026 test showed efficiency only in charts, did not describe how power was measured, and did not score answers.
Built with what exists
Raspberry Pi 5; Raspberry Pi AI HAT+ 2 with Hailo's models; Jetson Orin Nano Super Developer Kit; llama.cpp; a logging power meter at the wall. MLPerf's own method measures whole-system AC power at the wall with SPEC PTDaemon.
First step
Run Qwen2.5 1.5B in llama.cpp on a Raspberry Pi 5 plugged into a logging meter, and record watts, tokens per second and answers for 100 questions.
Done looks like
For each board and model, joules per question and the share answered correctly over at least 500 questions, idle power reported separately, three repeat runs within 5% of each other, and raw power logs and scripts published.
Open 02 · the brief
An outside Coralboard test
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
Compile the four MLPerf Tiny reference models, which detect keywords, detect whether a person is in view, sort small images into classes and spot abnormal machine sounds, with Synaptics' open Torq compiler, and run them on the Coralboard SL's NPU. For each, report accuracy against the same model run in LiteRT on a PC, and milliseconds and millijoules per inference. What makes it possible: Google's open Coral NPU design, and Synaptics' open compiler and board, on sale since 20 October 2026.
Why it is still open
Synaptics' release gives no power measurements, and its one speed figure (Gemma3 270M, "up to 12-16 tokens/second") names no tester. Synaptics is not among the nine organizations in MLPerf Tiny v1.4, published on 7 July 2026.
Built with what exists
Coralboard SL (Astra SL2619); Torq compiler v2.3.1, which accepts TFLite and ONNX models and has a profiler; the MLPerf Tiny reference models (Apache 2.0); LiteRT on a PC; a logging USB or bench power meter.
First step
Order the board, install Torq v2.3.1, and compile its documented MobileNetV2 example, then the MLPerf Tiny image model.
Done looks like
A table of four models, each with reference accuracy, board accuracy, the difference in percentage points, and milliseconds and millijoules per inference, measured on the full test sets, with compiler version and scripts published.
Open 03 · the brief
Blade defects on small boards
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
Liquid AI published a worked example in which its LFM3-VL-2B model is fine-tuned on WTBs2025, a set of wind-turbine blade photos (CC BY 4.0). Repeat that fine-tune, then apply quantization, storing the model's numbers with fewer bits, by converting it to 4-bit GGUF files; run it with llama.cpp on a Raspberry Pi 5 and a Jetson Orin Nano Super and score it on the same 1,372 held-out photos. The work uses photos only and never touches a working turbine. What makes it possible: small open vision models that can be fine-tuned on one GPU (LFM3, October 2026).
Why it is still open
Liquid's write-up reports 66.5% defect-type accuracy (95% range 64.1% to 69.2%), by its own measurement, after training on one A100 GPU, and gives no results for quantized files or for any edge board.
Built with what exists
Liquid's cookbook example; LFM3-VL-2B (LFM Open License v1.0), which its model card says to run with llama.cpp build b11960 or newer; one rented GPU; Raspberry Pi 5; Jetson Orin Nano Super.
First step
Run the cookbook's evaluation on the unmodified base model on a rented GPU and check that it reproduces the 9.2% starting accuracy.
Done looks like
Defect-type accuracy with a 95% range on all 1,372 photos for the full-precision model and for each 4-bit version on each board, plus seconds per photo, stating whether each range overlaps 64.1% to 69.2%.