Every row below exists because someone ran the bench on their own device and sent the result in.
iPhone19,7): qwen3.5-2b, minicpm5-1bMac16,7): qwen3-0.6b, qwen3-4b, qwen3.5-0.8b, qwen3.5-2b, minicpm5-1b, minicpm5-2b, qwen3.6-27b, qwen3.8-27biPhone18,3): minicpm5-1biPhone18,1): minicpm5-2bCommunity field data — measured by the Bench tab of the CoreAI Zoo app on contributors’ own devices and submitted as bench-result issues (the public audit log — closing an issue does not remove its row). The app measures and builds the result blob; no number in this table was typed by a human. This is NOT a controlled-environment benchmark — background load and heat show up here as real-world variance. For the maintainer’s controlled numbers see each model card.
One table per protocol. Cell = median across submissions of each submission’s
median warm decode tok/s; n = accepted submissions. Cells with n < 3 are
provisional (marked *). Blobs with Low Power Mode on or a serious/critical
thermal state before the run are excluded from medians (counted below). Model load
time is not published (the app’s load_s includes a first download). Devices are
the utsname.machine id; ids without a marketing name yet show raw.
Add your device: install CoreAI Zoo
from the App Store → Bench tab → pick a chat model → Run → Submit on GitHub. Your
device becomes a row here on the next aggregation
(python3 scripts/aggregate_bench.py).
Generated by scripts/aggregate_bench.py — do not edit by hand. Last run: 2026-10-02 13:39 UTC.
zoo-chat-v1 — decode tok/s (median warm)Protocol zoo-chat-v1 (live — CoreAI Zoo 2.0+, Bench tab): a fixed text prompt (“Explain, in detail, how a transformer language model generates text one token at a time…”, ~52–56 tokens depending on the tokenizer) → 256 greedy decode tokens, S=1 prefill (COREAI_CHUNK_THRESHOLD=1), 1 cold + 3 warm runs, session reset before every run so each run is a full prefill, measured through the kit ChatSession — the app’s real chat path. decode tok/s is the 32-token rolling-window steady-state rate; prefill tok/s = prompt tokens / TTFT.
| Device | qwen3-0.6b |
qwen3-4b |
qwen3.5-0.8b |
qwen3.5-2b |
minicpm5-1b |
minicpm5-2b |
qwen3.6-27b |
qwen3.8-27b |
|---|---|---|---|---|---|---|---|---|
iPhone 18 Pro Max (A20 Pro, iPhone19,7) |
— | — | — | 48.9* (n=1) | 101.2* (n=1) | — | — | — |
iPhone 17 (A19, iPhone18,3) |
— | — | — | — | 59.1* (n=1) | — | — | — |
iPhone 17 Pro (A19 Pro, iPhone18,1) |
— | — | — | — | — | 25.3* (n=1) | — | — |
MacBook Pro 16-inch 2024 (M4 Pro, Mac16,7) |
376.8* (n=1) | 71.0* (n=1) | 180.6* (n=1) | 99.1* (n=1) | 205.5* (n=1) | 90.4* (n=1) | 8.7* (n=1) | 8.6* (n=1) |
pb-random-v1 — decode tok/s (median warm)Protocol pb-random-v1 (July 2026 CoreAIChat TestFlight build, no longer produced — kept as the historical table): fixed 128-token random prompt (seed 0) → 256 greedy decode tokens, S=1 prefill (COREAI_CHUNK_THRESHOLD=1), 1 cold + 3 warm runs on a freshly created engine. Raw token ids fed to a forked engine, so its numbers are not byte-comparable with zoo-chat-v1.
| Device | qwen3.5-0.8b |
|---|---|
iPhone 17 Pro (A19 Pro, iPhone18,1) |
68.4* (n=1) |
Accepted submissions: 13 · excluded by environment filter: 0 · rejected (schema/protocol): 0