BRENNEN

Industries / Technology / Software and SaaS / Cloud and data centers / Edge computing

Running AI at the edge

This is the work of running trained AI models that see, hear or handle language on small computers on site, in a camera, a shop or a vehicle, instead of sending the data to a distant data center. It covers the accelerator chips and boards, the software that shrinks and runs the models, and the shared tests that measure them.

7

Milestones achieved, from 2014 to 2026.

3

To watch for: announced, not yet confirmed outside.

3

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 2014

    Graphics chips for small devices

    On 25 March 2014 NVIDIA announced the Jetson TK1, a small board built on its Tegra K1 chip, with 192 graphics cores that developers program in CUDA, NVIDIA's language for doing many sums at once. NVIDIA rated it at 326 gigaflops (billions of operations a second) and called it "the world's first mobile supercomputer for embedded systems", its own claim; BDTI, an outside analysis firm, reported preorders at $192.

    rss.globenewswire.combdti.com

    Credit

    NVIDIA

  2. 2017

    A common runtime for devices

    On 14 November 2017 Google released a developer preview of TensorFlow Lite, a runtime (the program that runs a trained model) for phones and embedded devices, with a core of about 70 KB. On 4 September 2024 Google renamed it LiteRT; by Google's count it then ran in more than 100,000 apps on 2.7 billion devices.

    developers.googleblog.comdevelopers.googleblog.com

    Credit

    Google's TensorFlow team

  3. 2017

    AI on a USB stick

    Movidius showed its Fathom stick in April 2016; after Intel bought the company, it went on sale on 20 July 2017 at $79 as the Movidius Neural Compute Stick. It holds a Myriad 2 vision chip and lets a small computer such as a Raspberry Pi run a trained network offline. Intel stopped shipping the second-generation stick on 30 June 2022 and ended technical support on 30 June 2023.

    LessonA plug-in accelerator lasts only as long as its maker supports it; Intel sent stick users to a different product.

    techcrunch.comintel.com

    Credit

    Movidius; Intel

  4. 2019

    A shared test for inference

    On 6 November 2019 the MLPerf group published its first inference results, inference meaning running a trained model: 595 results from 14 organizations, on systems ranging from "embedded devices and smartphones to large-scale data center systems". NVIDIA described two of its four test patterns, single-stream (one input at a time, as from one camera) and multi-stream (several inputs at a fixed rate), as the edge-focused ones. On 16 June 2021 MLCommons added MLPerf Tiny, for models of about 100 KB on very low-power chips.

    mlcommons.orgdeveloper.nvidia.comhpcwire.com

    Credit

    the MLPerf group, now MLCommons (paper by Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson and others)

  5. 2023

    A cloud-tied kit withdrawn

    Microsoft announced Azure Percept on 2 March 2021, a kit with a camera module built on Intel's Myriad X chip, tied to Azure cloud tools for building models and sending them to the device. Microsoft retired the kit, its audio accessory and the cloud services behind them on 30 March 2023, saying they had been in preview.

    LessonWhen the maker withdraws the cloud side, the device loses its updates and support, and users' cloud projects that were not closed kept billing.

    theregister.comlearn.microsoft.com

    Credit

    Microsoft

  6. 2025

    Robot computer runs language models

    On 25 August 2025 NVIDIA began selling the Jetson AGX Thor developer kit at $3,499, a computer for robots with 128 GB of memory that works within 130 watts. The same day the review site ServeTheHome measured 149.1 tokens (word pieces) per second on the Llama 3.1 8B language model, against the 150.8 NVIDIA expected.

    nvidianews.nvidia.comservethehome.com

    Credit

    NVIDIA; measured by ServeTheHome

  7. 2026

    Accelerator slower than its host

    Raspberry Pi has sold a $70 AI Kit with a Hailo-8L chip, rated at 13 TOPS (trillions of operations a second), since 4 June 2024. On 15 January 2026 it released the $130 AI HAT+ 2, with a Hailo-10H chip rated at 40 TOPS on 4-bit numbers and 8 GB of its own memory, sold for running small language models. In tests published that day, Jeff Geerling found the Raspberry Pi 5's own processor faster than the Hailo-10H on most models, with the Hailo close only on Qwen2.5 Coder 1.5B, though the Hailo was slightly more efficient in most cases.

    LessonA TOPS rating does not predict language-model speed; the Hailo is held to 3 W, while the Pi's processor may draw 10 W.

    raspberrypi.comjeffgeerling.comraspberrypi.com

    Credit

    Raspberry Pi and Hailo; tested by Jeff Geerling

Watch for

  1. Open NPU design and compiler

    An NPU (neural processing unit) is a processor built to run AI models. Google says its open Coral NPU design now has a matrix unit doing 256 multiply-accumulates per clock tick, "configurable for up to 1 TOPS and targeting 10mW average power for inference", and that its open Coral Compiler v1.0 takes JAX, PyTorch and TensorFlow models to the hardware. Google's own documents show the compiler's output running in simulation, while the Synaptics Coralboard SL that went on sale the same day uses Synaptics' own Torq compiler (see the Note "An open compiler").

    Achieved whena chip containing the new matrix unit is on sale and someone outside Google and Synaptics runs the compiler's output on it and measures power.

    developers.googleblog.com

    Claimed by

    Google Research, October 2026

  2. Small models that run faster

    Liquid AI says its LFM3-1.7B text model produces text 1.8 to 2.8 times faster than the next-fastest comparable model and uses the least memory, in a test with llama.cpp and 4-bit weights on three devices it does not name. It also says each of its six LFM3 models can be fine-tuned, trained further for one job, on a single GPU (see the Note "One GPU to fine-tune").

    Achieved whensomeone outside Liquid AI repeats the speed comparison on named hardware, such as a Raspberry Pi 5, with published settings, and gets a similar ratio.

    liquid.ai

    Claimed by

    Liquid AI, October 2026

  3. Agent tasks 6.4 times faster

    NVIDIA says its TensorRT Edge-LLM software finished MLPerf's new edge agentic test, 1,007 turns of tool-calling work with the Qwen3.6-27B model, on one Jetson AGX Thor in 24 minutes 36 seconds, 6.4 times faster than a llama.cpp reference run of 2 hours 37 minutes, with 87.94% function-calling accuracy. The comparison is NVIDIA's own, the two runs stored the model's numbers in different formats (NVFP4 against Q4_K_M), and no power figure was given.

    Achieved whensomeone outside NVIDIA runs both setups on one Thor under MLPerf's accuracy check and gets a similar ratio.

    developer.nvidia.com

    Claimed by

    NVIDIA, September 2026

Open

  1. 01

    Energy per answer, cheap boards

    A public table of energy per correct answer for two or three small open language models, such as Qwen2.5 1.5B and Llama 3.2 1B, on low-cost boards: a Raspberry Pi 5 alone, a Raspberry Pi 5 with the AI HAT+ 2, and a Jetson Orin Nano Super. Power is measured at the wall while each board answers a fixed set of questions with known answers. What makes it possible: open models of 1 to 2 billion parameters now run on boards costing $100 to $250, and ggml 1.0.0, the library under llama.cpp, gives a fixed software version to test against (see the Note "A first stable number for local AI").

    The brief
    Credit · unclaimed
  2. 02

    An outside Coralboard test

    Compile the four MLPerf Tiny reference models, which detect keywords, detect whether a person is in view, sort small images into classes and spot abnormal machine sounds, with Synaptics' open Torq compiler, and run them on the Coralboard SL's NPU. For each, report accuracy against the same model run in LiteRT on a PC, and milliseconds and millijoules per inference. What makes it possible: Google's open Coral NPU design, and Synaptics' open compiler and board, on sale since 20 October 2026.

    The brief
    Credit · unclaimed
  3. 03

    Blade defects on small boards

    Liquid AI published a worked example in which its LFM3-VL-2B model is fine-tuned on WTBs2025, a set of wind-turbine blade photos (CC BY 4.0). Repeat that fine-tune, then apply quantization, storing the model's numbers with fewer bits, by converting it to 4-bit GGUF files; run it with llama.cpp on a Raspberry Pi 5 and a Jetson Orin Nano Super and score it on the same 1,372 held-out photos. The work uses photos only and never touches a working turbine. What makes it possible: small open vision models that can be fine-tuned on one GPU (LFM3, October 2026).

    The brief
    Credit · unclaimed

Open 01 · the brief

Energy per answer, cheap boards

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public table of energy per correct answer for two or three small open language models, such as Qwen2.5 1.5B and Llama 3.2 1B, on low-cost boards: a Raspberry Pi 5 alone, a Raspberry Pi 5 with the AI HAT+ 2, and a Jetson Orin Nano Super. Power is measured at the wall while each board answers a fixed set of questions with known answers. What makes it possible: open models of 1 to 2 billion parameters now run on boards costing $100 to $250, and ggml 1.0.0, the library under llama.cpp, gives a fixed software version to test against (see the Note "A first stable number for local AI").

Why it is still open

MLPerf Inference v6.1, published on 16 September 2026, had 24 edge results from five organizations; the only board-sized system was the $3,499 Jetson AGX Thor kit, the rest desktops and servers, and the round's summary file marks none of its 492 results as carrying a power measurement. Jeff Geerling's January 2026 test showed efficiency only in charts, did not describe how power was measured, and did not score answers.

github.comjeffgeerling.com

Built with what exists

Raspberry Pi 5; Raspberry Pi AI HAT+ 2 with Hailo's models; Jetson Orin Nano Super Developer Kit; llama.cpp; a logging power meter at the wall. MLPerf's own method measures whole-system AC power at the wall with SPEC PTDaemon.

  1. First step

    Run Qwen2.5 1.5B in llama.cpp on a Raspberry Pi 5 plugged into a logging meter, and record watts, tokens per second and answers for 100 questions.

  2. Done looks like

    For each board and model, joules per question and the share answered correctly over at least 500 questions, idle power reported separately, three repeat runs within 5% of each other, and raw power logs and scripts published.

Open 02 · the brief

An outside Coralboard test

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

Compile the four MLPerf Tiny reference models, which detect keywords, detect whether a person is in view, sort small images into classes and spot abnormal machine sounds, with Synaptics' open Torq compiler, and run them on the Coralboard SL's NPU. For each, report accuracy against the same model run in LiteRT on a PC, and milliseconds and millijoules per inference. What makes it possible: Google's open Coral NPU design, and Synaptics' open compiler and board, on sale since 20 October 2026.

Why it is still open

Synaptics' release gives no power measurements, and its one speed figure (Gemma3 270M, "up to 12-16 tokens/second") names no tester. Synaptics is not among the nine organizations in MLPerf Tiny v1.4, published on 7 July 2026.

synaptics.commlcommons.org

Built with what exists

Coralboard SL (Astra SL2619); Torq compiler v2.3.1, which accepts TFLite and ONNX models and has a profiler; the MLPerf Tiny reference models (Apache 2.0); LiteRT on a PC; a logging USB or bench power meter.

  1. First step

    Order the board, install Torq v2.3.1, and compile its documented MobileNetV2 example, then the MLPerf Tiny image model.

  2. Done looks like

    A table of four models, each with reference accuracy, board accuracy, the difference in percentage points, and milliseconds and millijoules per inference, measured on the full test sets, with compiler version and scripts published.

Open 03 · the brief

Blade defects on small boards

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

Liquid AI published a worked example in which its LFM3-VL-2B model is fine-tuned on WTBs2025, a set of wind-turbine blade photos (CC BY 4.0). Repeat that fine-tune, then apply quantization, storing the model's numbers with fewer bits, by converting it to 4-bit GGUF files; run it with llama.cpp on a Raspberry Pi 5 and a Jetson Orin Nano Super and score it on the same 1,372 held-out photos. The work uses photos only and never touches a working turbine. What makes it possible: small open vision models that can be fine-tuned on one GPU (LFM3, October 2026).

Why it is still open

Liquid's write-up reports 66.5% defect-type accuracy (95% range 64.1% to 69.2%), by its own measurement, after training on one A100 GPU, and gives no results for quantized files or for any edge board.

raw.githubusercontent.com

Built with what exists

Liquid's cookbook example; LFM3-VL-2B (LFM Open License v1.0), which its model card says to run with llama.cpp build b11960 or newer; one rented GPU; Raspberry Pi 5; Jetson Orin Nano Super.

  1. First step

    Run the cookbook's evaluation on the unmodified base model on a rented GPU and check that it reproduces the 9.2% starting accuracy.

  2. Done looks like

    Defect-type accuracy with a 95% range on all 1,372 photos for the full-precision model and for each 4-bit version on each board, plus seconds per photo, stating whether each range overlaps 64.1% to 69.2%.