BRENNEN

Industries / Technology / Software and SaaS / AI and machine learning

Running models for users

This is the work of answering people's requests with a trained model, through an app, through a paid connection that other programs call (an API), or on the user's own computer, quickly and at low cost.

7

Milestones achieved, from 2016 to 2026.

2

To watch for: announced, not yet confirmed outside.

3

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 2016

    Translation served by neural networks

    Google moved all Chinese-to-English translations in the Google Translate app and website to a neural network, GNMT. By Google's own side-by-side ratings by people, GNMT cut translation errors by 55% to 85% on several major language pairs. Google said its own AI chips (TPUs) gave it enough computing power to serve the model fast enough.

    research.google

    Credit

    Quoc V. Le, Mike Schuster and the Google Brain team

  2. 2020

    Models sold through an API

    On 11 June 2020 OpenAI opened a private beta of an API: a program sends text and gets text back from a model of the GPT-3 family, and pays for the use. OpenAI said it would cut off "obviously harmful use-cases" and named the API as "a revenue source" for its work. Selling access by the request, with the model kept on the seller's computers, became the industry's main way to sell models.

    openai.com

    Credit

    OpenAI

  3. 2022

    A free chat window

    On 30 November 2022 OpenAI released ChatGPT, a chat version of its GPT-3.5 model, free to use as a "research preview". In a Pew Research Center survey of 5,123 US adults (24 February to 2 March 2025), 34% said they had used ChatGPT, about double the share in summer 2023, and 58% of adults under 30 had.

    openai.compewresearch.org

    Credit

    OpenAI; usage figures from Pew Research Center

  4. 2023

    Models on your own computer

    On 10 March 2023 Georgi Gerganov made the first commit of llama.cpp, a program in C and C++ that runs language models on ordinary computers. Its GGUF file format stores a model's numbers in compressed form (quantization): the common Q4_K type uses about 4.5 bits per number instead of 16, so a model needs far less memory. In November 2026 the project reached version 1.0.0, with support for more than 150 model designs, packages in Debian, Homebrew and Winget, and hardware support contributed by NVIDIA, AMD, Qualcomm, Arm, IBM and Intel (see the Note "A first stable number for local AI").

    github.comhuggingface.cohuggingface.co

    Credit

    Georgi Gerganov and the ggml-org contributors

  5. 2023

    Less memory wasted per user

    While a model answers, it keeps working notes for each conversation (the KV cache) in the chip's memory. A UC Berkeley team found that existing serving software wasted much of that memory, which limited how many users one chip could serve at once. Their method, PagedAttention, stores the notes in small blocks, the way a computer's operating system manages memory; by the authors' tests their software, vLLM, served 2 to 4 times as many requests at the same speed. By May 2025 vLLM had more than 1,000 contributors, ran on chips from NVIDIA, AMD, Google, AWS and Intel, and was taken in by the PyTorch Foundation.

    arxiv.orgpytorch.org

    Credit

    Woosuk Kwon, Zhuohan Li, Ion Stoica and colleagues, UC Berkeley

  6. 2025

    Same score, falling price

    Epoch AI, a research group that tracks AI, measured how fast the price of reaching a given test score falls. The price of matching GPT-4's score on a set of PhD-level science questions fell about 40 times per year; across six tests the rate ranged from 9 to 900 times per year. One example from its tables: GPT-4 cost $37.50 per million tokens (chunks of text) for 86.0 on the MMLU test in March 2023, and Gemini 2.0 Flash cost $0.18 for 88.0 in February 2025. Epoch notes that the fastest drops came in the latest year, so it is less clear they will last.

    epoch.ai

    Credit

    Ben Cottier, Ben Snodin, David Owen and Tom Adamczewski, Epoch AI

  7. 2026

    Same model, different answers

    Many companies rent out the same free-to-download model, and buyers assumed they got the same thing. In August 2025 Artificial Analysis, a firm that benchmarks AI services, ran gpt-oss-120b at 11 providers on a math competition test (AIME 2025): scores ran from 36.7% to 93.3%, and Microsoft traced Azure's 80% to an old serving program that ignored the requested reasoning effort. In August 2026 the firm launched its Endpoint Accuracy Index, which compares each provider with the official model run by the firm itself. It found some gpt-oss-120b providers scoring 22% on a tool-use test against 37% for the reference, and blamed tight output limits, shortened context and differences in reasoning effort.

    LessonThe same model name does not guarantee the same model; buyers have to test the endpoint they pay for.

    simonwillison.netartificialanalysis.ai

    Credit

    Artificial Analysis

Watch for

  1. Energy per prompt, self-measured

    Google says the median text prompt in its Gemini apps used 0.24 watt-hours of energy, emitted 0.03 grams of carbon dioxide equivalent and used 0.26 milliliters of water, counting idle machines, host computers and data-center overhead. It says energy per median prompt fell 33 times between May 2024 and May 2025. Google's own footnote says "The data and claims have not been verified by an independent third-party."

    Achieved whena party outside Google audits the figures or measures a comparable service with the same boundary.

    cloud.google.com

    Claimed by

    Amin Vahdat and Jeff Dean, Google, August 2025

  2. Restarts in seconds

    Starting a large model from nothing on a fresh machine (a cold start) took 6 to 19 minutes for the largest models in NVIDIA's tests. NVIDIA says its open-source Snapshot saves a running model service and brings it back in 3.6 to 42.7 seconds, 8 to 36 times faster, on its B200 chips with 1 to 8 chips per service (see the Note "Faster than a cold start"). Two design partners, Crusoe and Saturn Cloud, tried it on their own work.

    Achieved whena company outside NVIDIA and its design partners publishes restore times on its own services.

    developer.nvidia.com

    Claimed by

    NVIDIA, November 2026

Open

  1. 01

    Cold starts timed alike

    A public, repeatable test of how long each rented "serverless" GPU service (one that shuts a model down when idle and starts it on demand) takes to give its first answer after sitting idle. Every service runs the same open model on the same class of chip, triggered the same way, on a schedule.

    The brief
    Credit · unclaimed
  2. 02

    Quant quality, model by model

    A standing table of how far each popular compressed (quantized) model file strays from the original model, for the 20 most-downloaded open models in GGUF format and every major uploader of them. People running models on their own computers pick among dozens of such files with no common measure.

    The brief
    Credit · unclaimed
  3. 03

    A page's price per language

    A public table of what the same text costs to send through each of the ten most-used paid model APIs in each of about 200 languages, compared with English. APIs charge per token, and the same sentence splits into more tokens in some languages than others. The table is refreshed whenever a provider changes its model or prices.

    The brief
    Credit · unclaimed

Open 01 · the brief

Cold starts timed alike

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public, repeatable test of how long each rented "serverless" GPU service (one that shuts a model down when idle and starts it on demand) takes to give its first answer after sitting idle. Every service runs the same open model on the same class of chip, triggered the same way, on a schedule.

Why it is still open

Published comparisons repeat the vendors' own numbers. A HostFleet comparison of eight GPU services, updated November 2026, states "HostFleet did not benchmark cold starts", and a 2026 comparison of nine providers on DEV Community says "I collected documented numbers". NVIDIA's Snapshot figures come from NVIDIA's own tests.

hostfleet.pages.devdev.to

Built with what exists

paid accounts, used within their terms, on services such as Modal, RunPod, Baseten, Replicate, Koyeb and Google Cloud Run; one open model at two sizes (about 8 billion and 70 billion parameters) served with vLLM; a Python script that waits past each service's idle timeout, sends one request and records the time to the first token; GitHub Actions to run it on a schedule.

  1. First step

    Deploy one 8-billion-parameter model on two services, set both to shut down when idle, and time 20 cold starts on each.

  2. Done looks like

    eight services, two model sizes, at least 50 cold starts each; median and 95th-percentile times to first answer published with the code and raw logs; the same run repeated a month later.

Open 02 · the brief

Quant quality, model by model

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A standing table of how far each popular compressed (quantized) model file strays from the original model, for the 20 most-downloaded open models in GGUF format and every major uploader of them. People running models on their own computers pick among dozens of such files with no common measure.

Why it is still open

The closest work takes one model at a time. In April 2026 LocalBench measured 53 compressed versions of Gemma 4 31B from four uploaders against the full-precision model and found that "ggml-org and lmstudio-community quants never appear on the Pareto frontier" (the set of best trade-offs between size and accuracy) except at Q8_0; the same month it measured 87 files of Qwen 3.6 27B from seven uploaders. Its posts do not say whether the code and test text are released, and no standing table across many models was found.

localbench.substack.comlocalbench.substack.com

Built with what exists

llama.cpp's perplexity tool, which compares a compressed model's predictions with the original's using KL divergence (a measure of how far two sets of probabilities differ); the Hugging Face Hub API to list GGUF files and download counts; one rented GPU to make the full-precision references; a test text mixing code, chat, non-Latin scripts and long documents; one task test, such as a tool-use benchmark.

  1. First step

    Reproduce LocalBench's Gemma 4 31B numbers for two compression levels on one rented GPU.

  2. Done looks like

    20 models, every compression level from the three largest uploaders, a KL divergence and one task score for each file; code and raw numbers public; new uploads of covered models measured within a week.

Open 03 · the brief

A page's price per language

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public table of what the same text costs to send through each of the ten most-used paid model APIs in each of about 200 languages, compared with English. APIs charge per token, and the same sentence splits into more tokens in some languages than others. The table is refreshed whenever a provider changes its model or prices.

Why it is still open

Ahia and colleagues found in 2023 that, on OpenAI's API, speakers of many of 22 languages were "overcharged while obtaining poorer results". An August 2026 audit covered five languages and three tokenizers. An open project from July 2026 measured 10 downloadable tokenizers in all 200 FLORES-200 languages, but it counts only tokenizers that can be downloaded, so paid models such as Anthropic's Claude and Google's Gemini are missing, and it reports tokens, not prices.

arxiv.orgarxiv.orggithub.com

Built with what exists

the FLORES-200 test set from Meta (1,012 sentences, the same content in every language); the tiktoken and Hugging Face tokenizers libraries for open tokenizers; the token-counting endpoints that Anthropic and Google offer for their closed ones; each provider's published price list; a static web page.

  1. First step

    Fork the July 2026 project and add one closed model through its token-counting endpoint for 20 languages.

  2. Done looks like

    about 200 languages times 10 paid APIs, each cell the cost of the 1,012 sentences relative to English, reproducible from a published script; refreshed at least three times in six months.