How we measure — exactly.

The quality axis

Quality runs on EleutherAI’s lm-evaluation-harness, using its leaderboard task group — the frozen task set behind the Open LLM Leaderboard v2 (retired March 2025), so our scores remain comparable to that archived record:

• Coding is added as HumanEval + MBPP (pass@1, greedy), because on-device assistants are so often coding assistants.
• Caps and seeds. Each task is capped at 500–1,000 items with a fixed, published seed, so a full sweep stays affordable to reproduce. Per-task standard error ships with every score; per-sample logs ship with every self-run result.
• Think-mode models run twice. Dual-mode models (Qwen3, SmolLM3) are evaluated in think and no-think modes separately and reported separately — never averaged. The mode is labeled on the score.
• The composite. The Small-Model Intelligence Score is a transparently weighted aggregate of the tasks above with bootstrapped confidence intervals. It exists only as a self-run number — we do not synthesize a composite out of other people’s single-task figures. Until our harness pass lands for a model, its row leads with a single cited benchmark, named exactly.
• The rolling slice. A LiveBench-style contamination-resistant slice — recent, objectively-scored questions refreshed on a schedule — runs alongside the frozen set as the anti-gaming anchor, and is reported separately from it.

The efficiency axis

Efficiency is measured per model × runtime × device, at batch size 1 — the shape of real on-device use. Native GGUF throughput comes from llama-bench; browser runs are instrumented in the page. We report:

• Decode throughput. tokens/second generating, batch size 1 — the number you feel while it types.
• Prefill throughput. tokens/second ingesting the prompt.
• Time to first token. ms from request to first output token.
• Peak RAM / VRAM. measured runtime footprint — NOT the file size. Running a model needs materially more memory than its weights (~2× is a working rule); we publish only what we measure.
• On-disk size. the weight file at the exact labeled precision.
• Cold start. time from launch to first token on a cold cache. Critical in-browser, where a multi-hundred-MB download dominates the experience.
• Energy per response. Joules for the standard workload, idle power subtracted.
• Average / peak power. Watts during decode; average draw feeds Intelligence-per-Watt.

• Standard workload. 512 input / 128 output tokens, plus 1k-input and 10k-input variants for context scaling. Token counts are normalized to one reference tokenizer so “tokens/second” means the same thing across model families.
• Medians over ≥5 runs. Every efficiency figure is the median of at least five repeats, published with its spread. A single run is never presented as a settled number — the loader rejects self-run records with fewer than three repeats, and our own bar is five.
• Thermal and power state disclosed. Plugged in vs. battery, power profile, and ambient conditions are part of the record — Apple Silicon in particular throttles on battery, and a number without its power state is not reproducible.
• Definitions aligned with MLPerf Client (MLCommons) where they overlap, so our TTFT and tok/s can be sanity-checked against an industry benchmark. Energy methodology follows the framing of Stanford’s Intelligence per Watt work: per-response energy with idle draw subtracted.

Quantization policy

The exact precision is labeled on every row, and it applies to the whole row. The apples-to-apples backbone for native and wllama runs is Q4_K_M GGUF; browser MLC runs use the MLC 4-bit format and are labeled as such; rows sourced from publisher reporting carry the released precision (usually bf16). Where we publish a quality delta across precisions — the FP16 → Q8 → Q4 view — it is our own measured drop for that model, run through the same harness at each precision. We do not assert generic “quantization costs X%” ranges; the honest answer is a measured number per model, and the striking fact that almost nobody publishes that number is one reason this board exists.

Runtimes

Runtimes are columns, not competitors — a model is measured across the runtimes people actually use. Native desktop: llama.cpp (GGUF) and Ollama (a desktop llama.cpp-based GGUF runtime — it is not a browser technology). In-browser: WebLLM running MLC-compiled models over WebGPU (not GGUF), transformers.js running ONNX via ONNX Runtime Web (WebGPU where available — still experimental there — with a WASM fallback), and wllama, which runs unmodified GGUF through a WASM build of llama.cpp. Runtime name and version are part of every record.

Devices

Every self-run record names its device exhaustively: machine, chip, RAM, OS version, and the driver or browser build the run went through, plus the power state. There is no “a laptop”. As of the current preview no device runs have landed yet — the device list on the board is therefore empty, and that emptiness is displayed rather than papered over. The first devices will be the machines this lab physically owns, named exactly.

Provenance rules

• measured — produced by this lab. The record embeds the device, the full config lockfile (harness/runtime versions, flags, seeds, model file), the date, and repeat count.
• cited — copied from a primary source: the model card, the paper, or the publisher’s file listing. The record carries the URL we verified and the retrieval date. If the source is not live and primary, the number does not go on the board.
• pending — not yet measured. Rendered as “—”, excluded from rankings, never zero.
• These rules are enforced in code: the results loader fails the build on an unsourced external value, a self-run record without its device and lockfile, or a “pending” group that somehow contains values.

Contamination stance

Frozen benchmarks leak into training data; every static suite decays. Our stance: keep the frozen leaderboard set for comparability with the archived record, run the rolling slice for freshness, report both, and date-stamp everything. When a model’s frozen-set score and rolling-slice score diverge sharply, that divergence is itself published — it is signal, not noise to smooth over.

Where everything lives

Result records are public JSON at lib/labs/results/ in the site’s repository — one file per model × precision × runtime × device, each embedding its config lockfile, and each rendered verbatim in the “full records” section of the leaderboard. The machine-readable board at /labs/leaderboard.json is generated from the same loader as the page, so the two cannot drift. Self-run results ship with their per-sample harness logs.

Tooling we build on

We build on existing, verifiable tooling and say so — none of it is ours, all of it is load-bearing:

• lm-evaluation-harness (EleutherAI) — Quality axis — we run its `leaderboard` task group (the frozen Open-LLM-Leaderboard-v2 tasks) so scores stay comparable to the archived record.
• llama-bench (llama.cpp) — Native prefill/decode throughput for GGUF runs — average tok/s with standard deviation, sampling and tokenization excluded.
• WebLLM (MLC) — In-browser WebGPU runtime — models compiled to the MLC format (not GGUF).
• MLC-LLM — The compilation stack behind WebLLM and its mobile targets.
• transformers.js — In-browser ONNX runtime via ONNX Runtime Web — WebGPU where available (still experimental there), WASM fallback.
• wllama — GGUF in the browser via a WASM build of llama.cpp — the browser path for unmodified GGUF files.
• Ollama — Desktop GGUF runtime (llama.cpp-based) — what people actually run locally. A desktop column, never a browser one.
• MLPerf Client (MLCommons) — Where possible we align metric definitions (TTFT, tok/s) with MLPerf Client so numbers are cross-checkable. Note it ranks hardware on fixed models — the inverse of this board.
• LiveBench — The model for our rolling contamination-resistant slice — monthly-refreshed questions with objective ground truth.
• Intelligence per Watt (Stanford Hazy Research / Scaling Intelligence Lab) — The energy-measurement precedent — per-query accuracy plus energy profiling across local models and accelerators (arXiv:2511.07885). We adopt its framing for energy-per-response.