Why an honest on-device model leaderboard does not exist yet — and what we are building

The benchmark world splits into two camps that never meet — quality-but-cloud-only, and efficiency-but-ranks-hardware. The models people actually run on their own devices fall in the gap. Here is the gap, mapped, and the board we are building to fill it.

Ask a simple question: what is the smartest language model that actually runs well on the laptop you already own?

You would expect a well-maintained table to answer that. There isn't one. After surveying what exists (every claim below is linked to the source we checked), the field splits cleanly into two camps — and the question falls between them.

Camp one: quality, measured in the cloud

Artificial Analysis is the reference for model quality and API performance, and it is good at what it does. But what it does is measure hosted endpoints — its own methodology describes benchmarking "AI inference services and systems," and its speed numbers are the throughput of a provider's API, not of your hardware. Small models do appear there (models in the 1.6B–4B class are listed), but only as cloud endpoints. There is no memory column, no energy column, no cold-start column — those concepts do not exist for a hosted API, and that is precisely the point: the on-device axis is structurally outside its model of the world. This is a gap, not a criticism, and we are not affiliated with them in any way.

The HuggingFace Open LLM Leaderboard — the open-weights quality reference for two years — retired in March 2025. Its evaluation tasks live on in EleutherAI's lm-evaluation-harness (more on why that matters below), but the maintained board itself is gone. Crowd-preference arenas measure something valuable, but it is preference, not on-device viability.

Camp two: efficiency, ranking your hardware instead of the model

MLPerf Client measures LLM performance on personal computers rigorously — time to first token, tokens per second — but it holds a small set of models fixed and ranks the hardware. LocalScore, a Mozilla Builders project, does the same in spirit: download it, benchmark your machine. Both answer "how fast is this laptop?", not "which model should I put on it?" The HF LLM-Perf Leaderboard ranks models on latency, memory, and energy — on server GPUs (A100-class hardware, cloud-shaped batching), and its last update as we write this was a year old.

The one project that merged the axes — and stopped

The Edge LLM Leaderboard (NyunAI) had the right idea: 130+ sub-8B models, quantized GGUF, measured with llama.cpp's own benchmark tool on real edge hardware — prefill, decode, on-disk size. It is also, as of this writing, untouched since December 2024, runs on a single device (a Raspberry Pi 5), and reports no memory or energy measurements. The idea was validated and then abandoned. Nobody picked it up.

And the browser? Nobody. At all.

Here is the strangest part. WebGPU now ships enabled by default in Chrome and Edge, landed in Firefox on Windows in mid-2025, and shipped in Safari 26 (caniuse). Three real in-browser LLM runtimes exist — WebLLM running MLC-compiled models on WebGPU, transformers.js running ONNX, and wllama running GGUF through WASM. A model that runs in a browser tab is the lowest-friction deployment there is: no install, no driver, no cloud bill, no data leaving the machine.

We looked for a leaderboard that ranks generative LLMs in the browser — cold-start included, because a multi-hundred-megabyte first download dominates that experience. We did not find one. If you know of one, tell us and we will link it here. As far as we can tell, this table has an audience and no author.

What we are building

The Synthiq Labs leaderboard is our attempt at the merged table, built on three commitments:

Both axes, one board. Quality (via the same frozen task set the archived Open LLM Leaderboard used, so scores stay comparable to that record) and measured on-device efficiency: decode speed, time to first token, peak RAM, on-disk size, cold start, and energy per response — per model, per runtime, per named device, at batch size 1.
Dual provenance on every number. Every figure is tagged either measured — run by us, with the full config embedded in the public record — or cited, with a live link to the primary source. A cell we have not measured says pending. It never says zero, and it is never quietly filled in. Right now, at launch, that means something visible: the quality column is cited and every efficiency cell is pending. The board looks like a preview because it is one, and it will fill in run by run, in public.
The lead metric is Intelligence-per-GB. The question that matters on a device is not "what is the best model?" but "what is the smartest thing that fits in the space I have?" Quality divided by on-disk size — the efficiency frontier, drawn as a curve you can read directly.

Every number is a git commit. The result files are public JSON with the configuration inside; fork the repo, re-run the measurement, and if you get a different number, that is a bug report we want.

The methodology is published in full — item caps, seeds, repeat counts, thermal disclosure, quantization policy, contamination stance. The research itself ships as videos on the channel and as notes here.

Next note: how we measure, in plain English.