Synthiq Labs · open research

The efficiency frontier for models that run on your device.

Small models, benchmarked where they actually run. Every number tagged measured or cited — never invented.

See the leaderboard →How we measure

Highlights

9 models · 0 devices measured · preview

Intelligence per GB

Cited quality ÷ on-disk size · Higher is better

  1. 29.7Qwen3-0.6B
  2. 20.0Llama 3.2 1B-Instruct
  3. 15.9Qwen3-1.7B
  4. 9.9Llama 3.2 3B-Instruct
  5. 9.6Qwen3-4B

Quality

Publisher-cited MMLU-family · variant per record

  1. 77.3Qwen3-4B
  2. 67.3Phi-4-mini-instruct
  3. 64.4Qwen3-1.7B
  4. 63.4Llama 3.2 3B-Instruct
  5. 49.3Llama 3.2 1B-Instruct

On-disk size

bf16 weights · Lower is better

  1. 1.5 GBQwen3-0.6B
  2. 2.0 GBGemma 3 1B-it
  3. 2.5 GBLlama 3.2 1B-Instruct
  4. 4.1 GBQwen3-1.7B
  5. 6.2 GBSmolLM3-3B

Quality is publisher-reported (cited, linked per record); efficiency columns are pending our device runs. Pending is “—”, never zero.

Quality vs. size — the Pareto frontier.

Full board →
Quality vs. on-disk size — the Pareto frontier304050607080900123456789ON-DISK SIZE (GB) — PRECISION AS LABELED PER ROWQUALITY (%)Qwen3-0.6B (bf16) — MMLU-Redux: 44.6 · 1.5 GB · frontier · citedQwen3-0.6BLlama 3.2 1B-Instruct (bf16) — MMLU: 49.3 · 2.47 GB · frontier · citedLlama 3.2 1B-InstructQwen3-1.7B (bf16) — MMLU-Redux: 64.4 · 4.06 GB · frontier · citedQwen3-1.7BLlama 3.2 3B-Instruct (bf16) — MMLU: 63.4 · 6.43 GB · citedLlama 3.2 3B-InstructQwen3-4B (bf16) — MMLU-Redux: 77.3 · 8.04 GB · frontier · citedQwen3-4BPhi-4-mini-instruct (bf16) — MMLU: 67.3 · 7.67 GB · frontier · citedPhi-4-mini-instruct
measured cited Pareto frontierClick a point for its full record. The table below is the same data as text.

Public records

One JSON file per model × precision × runtime × device, config embedded. Fork it, re-run it.

Provenance on everything

Measured (named device + lockfile) or cited (live primary source). Unmeasured cells say pending.

A comparable ruler

Quality uses the frozen task set behind the archived Open LLM Leaderboard; efficiency is batch-1 medians.

The full method → · Raw JSON

Changelog

Derived from record dates · never hand-typed

  1. Jul 23resultGemma 3 1B-it (bf16) — record added (cited quality)
  2. Jul 23resultGemma 3 4B-it (bf16) — record added (cited quality)
  3. Jul 23noteHow we measure (a preview)
  4. Jul 23resultLlama 3.2 1B-Instruct (bf16) — record added (cited quality)
  5. Jul 23resultLlama 3.2 3B-Instruct (bf16) — record added (cited quality)
  6. Jul 23resultPhi-4-mini-instruct (bf16) — record added (cited quality)
  7. Jul 23resultQwen3-0.6B (bf16) — record added (cited quality)
  8. Jul 23resultQwen3-1.7B (bf16) — record added (cited quality)

+ 3 earlier entries — every record is on the board.

Lab Notes

All notes →
Jul 23, 2026

How we measure (a preview)

Read →Jul 23, 2026

Why an honest on-device model leaderboard does not exist yet — and what we are building

Read →

The research ships as videos first — watch the builds on the channel.

YouTube — Synthiq Labs →

Follow the research

New results and notes, when they land.

A short note when new results or write-ups publish — plus the videos on YouTube. No sequence, no sales follow-up.

Want this kind of engineering on your own product? That is the other side of the house — the studio.