The leaderboard is only worth anything if you can trust every cell in it. This note is the plain-English version of how each cell gets there; the full methodology has every knob, seed, and citation.
Quality: reuse the ruler everyone already knows
We do not invent a quality benchmark. We run EleutherAI's lm-evaluation-harness using its leaderboard task group — the exact frozen task set that powered the (now archived) HuggingFace Open LLM Leaderboard v2: MMLU-Pro, GPQA, BBH, MuSR, IFEval, and MATH level 5. Reusing a frozen public ruler means our scores are comparable to a large archived record, and anyone can re-run them with one command.
Three honesty details that most reporting skips:
- Capped and seeded. Each task is capped at 500–1,000 items with a fixed seed, so reproducing a full sweep is affordable, and every score ships with its standard error.
- Think-mode models get two rows. Models with a reasoning toggle (Qwen3, SmolLM3) score very differently with it on versus off. We run both modes and publish both, labeled. Averaging them would manufacture a number no user ever experiences.
- A rolling slice against contamination. Frozen benchmarks leak into training data over time. Alongside the frozen set we run a LiveBench-style rolling slice of recent, objectively-scored questions — and when a model aces the frozen set but stumbles on the fresh one, we publish the gap instead of smoothing it.
Efficiency: batch size 1, medians, on hardware we name
Cloud benchmarks batch requests; your laptop serves exactly you. Every efficiency number on the board is measured at batch size 1 on a named device — machine, chip, RAM, OS, driver or browser build, and power state, because a MacBook on battery is a slower computer than the same MacBook plugged in.
We report the things you feel: decode speed (how fast it types), time to first token (how long you wait), peak RAM (whether it fits at all — a model's file size is not its memory footprint; running one takes materially more than the weights), on-disk size, cold start (in the browser, the first download is the experience), and energy per response in joules, with idle draw subtracted.
Every figure is the median of at least five runs, published with its spread. One run is an anecdote; five are a measurement.
The tags: measured, cited, pending
Every number on the board carries one of three tags:
- measured — we ran it. The public record embeds the device and the full configuration.
- cited — a publisher reported it. The record links the primary source and the date we verified it.
- pending — nobody has measured it yet, least of all us. The cell shows a dash. It is never zero, and it is never a guess.
At launch the board is honest about its own state: quality figures are cited from model publishers, and the efficiency columns — the ones this project exists for — are pending until our runs land. Watching cited turn into measured, model by model, is watching the project happen.
Reproducibility is the product
Every result is a flat JSON file in the site's public repository, one per model × precision × runtime × device, with the exact configuration inside it. Every number on the board is a git commit. Fork it, re-run it, and if your number disagrees with ours, we want the bug report.
That is the whole trick. There is no proprietary harness, no adjustment layer, no editorial thumb on the scale — just published measurements you can check. The methodology page has the rest: quantization policy, tokenizer normalization, contamination stance, and the full tooling list with links.