A state and a typed question in, an answer with its probability out — nothing generated. The model scores one prompt and the answer is read at its answer slot: choice (which of these options), score (where on this scale), noul (yes or no, as P(yes)). One line:
let a = try await CoreAI.decide(
text,
["reply": .noul("Does the speaker want a response from the assistant?"),
"topic": .choice("What is the message about?", ["billing", "delivery", "other"]),
"urgent": .score("How urgent is it?", levels: ["can wait", "this week", "today"])])
a["reply"]?.noul // P(yes)
a["topic"]?.choice // "delivery"
a["urgent"]?.score // expected level, 0…2
The same questions about an image go to decider-2b-vision, every question of a call read in one
pass (measured below):
let b = try await CoreAI.decide(
image: frame, "You control the right paddle. The image shows the current game screen.",
["move": .choice("What should you do right now?", ["move paddle up", "move paddle down", "stay"])])
b["move"]?.choice // "move paddle up"
Eleven screens, each one a whole use: one action, the complete result. The first ten share one loaded text model; Room check loads the image model instead. The same sources build for the Mac and for the iPhone. Drive through Typing are the shapes the most-viewed System One posts of September 2026 use — a game driven by the model, bulk classification of rows, a natural-language permission gate for a coding agent, context compression by relevance, as-you-type reading — with the model on the device:
| Screen | What you do → what you get | Shape |
|---|---|---|
| Autofill | Copy an email or a message (⌘C in Mail is enough; Paste on the iPhone) → every field of a checkout form (HTML, in a web view) fills at once. The text is prefilled once; one question per field picks the line that holds it — “which line has the street of the address the order should be shipped to?” — as a choice among the text’s lines, then the line is trimmed to what the field takes. A copied single piece goes to its one field; a secret key is refused. | choice, 7 per email |
| Checklist | Open a contract (text, Markdown, PDF, or a photo of one) → a list of verdicts to your questions — noul: / choice: / score: lines you edit. Read once, answered N times. |
noul + choice + score |
| Sorter | Open a folder → what needs you first (a payment, a reply by a date), then everything filed by folder; Move files does it. Every file read once, two decisions each. | choice + choice |
| Search | Query × passages: one decision per passage, ranked by its probability — a reranker made of a chat model, no index. | noul or score |
| Speech gate | Record, split into utterances, transcribe each on device (Apple’s recognizer, no download), and ask one question per utterance: is this a request for the assistant, or a remark? Only the ones that pass would go to a language model. | choice |
| Drive | Press Drive → the model drives a car down a three-lane road. Every tick it reads the lanes it can reach — “left lane: rock 1 ahead”, “middle lane: clear” — and picks one; the road advances; rocks come. The tick rate is the decision rate. | choice, one per tick |
| Columns | Open a CSV (or paste one text per line), write the columns as questions → every row is answered: read once, one decision per column; sort by any column, save the CSV with the new columns. | choice / score / noul per column |
| Command guard | Paste the commands an agent wants to run, with a policy in plain words → each one comes back run / ask the user first / refuse, the refused ones on top. hooks/claude-code-guard.sh runs the same decision as a Claude Code PreToolUse hook. |
choice, one per command |
| Context | An agent transcript’s tool results, scored against the question being answered → the unrelated ones drop out of the context; the header says how many tokens went. | score, one per tool result |
| Typing | Type → at every pause three chips read the text so far: its tone, what you are doing, the emoji that fits (tap to insert). | score + choice + choice per pause |
| Room check | Press Check all over a folder of room photos (the PNGs in Documents/rooms, or -rooms <dir>) → every room READY, RECLEAN or CHECK. Each photo gets five questions — bed made, towels on the bed, clothes or trash on the floor, lamp on, suitcase left — read in one pass by decider-2b-vision; an answer below 0.8 goes to a person. |
choice × 5 per photo, one pass |
The two Shortcuts actions (“Ask yes/no”, “Classify text”) run the same decisions on any text without opening the app.
decide-cli serve (or systemone serve, the same server as one Homebrew-installed binary:
brew install john-rocky/tap/systemone) puts the loaded model behind a /v1/systemone
endpoint on this machine, in
the request and answer forms of the hosted System One API — state (a string, or structured
data), model, questions keyed by your ids with type / instructions / criteria;
answers back with choice and every option’s probability, score with its legend, noul as
the probability the statement holds, confidence, usage. A client written for the hosted
endpoint is pointed at this one by its base URL and nothing else changes:
swift run -c release decide-cli serve --model minicpm5-2b # http://127.0.0.1:8090/v1/systemone
swift run -c release decide-cli serve --backend fm # the same over Apple's on-device foundation model (one-hot answers; see below)
clients/systemone.sh # one request with curl
python3 clients/systemone.py # the same from Python, standard library only
TYPESAFE_BASE_URL=http://127.0.0.1:8090 TYPESAFE_API_KEY=local python3 your_client.py # the official SDKs; other clients have a base-URL setting
python3 conformance/check.py http://127.0.0.1:8090 # 22 requests in the hosted forms, every answer's shape checked
{"state": "Help! My payouts have been failing for 3 days.", "model": "minicpm5-2b",
"questions": {"is_urgent": {"type": "noul", "instructions": "Does this convey urgency?",
"criteria": {"true": "explicitly time-sensitive", "false": "no urgency expressed"}},
"queue": {"type": "choice", "instructions": "Which queue should handle this?",
"criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages", "other": "everything else"}}}}
{"model": "minicpm5-2b",
"answers": {"is_urgent": {"type": "noul", "noul": 0.7046, "confidence": 0.7046},
"queue": {"type": "choice", "choice": "billing", "probabilities": {"billing": 0.5078, "technical": 0.4492, "other": 0.0430}, "confidence": 0.2364}},
"usage": {"input_tokens": 239, "output_tokens": 2}, "timing_ms": 103.5733}
The probabilities are read at the catalog temperature (2.93 for minicpm5-2b; raw, the same
request answers noul 0.9274 and billing 0.5886). The state is prefilled once per request and every question rewinds to it; that request took
204 ms end to end on the Mac (M4 Max, 2026-09-23), four questions on a 69-token state 436 ms.
A third-party client library written for the hosted endpoint (system-one 0.1.0 on PyPI,
HTTPConfig(base_url=…)) got its typed answers back from this server unchanged, 179 ms round
trip for three questions.
So did the official TypeSafe Python SDK (typesafe-sdk 0.7.1) with only TYPESAFE_BASE_URL
set and any string as the key: 158–201 ms for three questions, models.list() included
(2026-09-23); the JavaScript SDK reads the same variable.
A structured state or instructions (an object or an array) is serialized the way Python’s
json.dumps(…, ensure_ascii=False) writes it, key order kept, so the model reads the bytes a
Python client would have sent (JSONValue in the kit). confidence is 1 − normalised entropy
of the distribution for a choice or a score and max(p, 1 − p) for a noul; probabilities are
rounded to four decimals. A choice lists up to the hosted API’s 255 options where the model
reads that many (minicpm5-2b does; the count per model).
A longer list comes back as a 422 that names the model’s count. GET /v1/models lists the
loaded model in the hosted form (models, each
name / description / release_date, the pinned revision beside them; the OpenAI-style
data list follows), GET /health answers ok, CORS is open so a page or a browser extension
can call it, and --host 0.0.0.0 serves the local network (a phone on the same Wi-Fi, another
machine). conformance/check.py <base_url> is the same contract as a test: 22 requests —
every shape above, the 16-, 17-, 255- and 256-option edges, structured state and instructions, a
three-message chat as the state, four malformed requests — and the keys and types each
answer must come back with; cases.json is the list, and any other /v1/systemone server
can be run against it. The codec is public API (SystemOne.request(from:),
SystemOne.response(model:answers:)) for an app that wants to accept or emit the form itself,
and CoreAI.systemOne(json:) answers a whole request in-process — the same reply this server
writes, with the typed answers beside it.
The same decision as a hook. hooks/claude-code-guard.sh is a Claude Code PreToolUse
hook: every Bash command the agent is about to run goes through decide-cli under the policy
at the top of the script, and “ask the user first” / “refuse” become the hook’s
permissionDecision. Build decide-cli once, point the hook at it in settings.json
(the script’s header shows the entry), and the gate runs on your machine — each call loads the
model (a few seconds with the weights cached) and decides in about 70 ms.
The same decisions as MCP tools. decide-cli mcp (or systemone mcp, the same server in
the Homebrew-installed binary) is a Model Context Protocol server on stdin/stdout with two
tools: decide (the /v1/systemone request above as its arguments, the response as
structuredContent and as text) and models (the catalog ids that can answer). Register the
built binary once and the agent’s own sessions see the tools:
claude mcp add systemone -- "$PWD/.build/release/decide-cli" mcp # Claude Code (add --scope user for every project)
codex mcp add systemone -- "$PWD/.build/release/decide-cli" mcp # Codex (writes [mcp_servers.systemone] in ~/.codex/config.toml)
{"mcpServers": {"systemone": {"command": "/path/to/decide-cli", "args": ["mcp"]}}}
(Cursor: that JSON in ~/.cursor/mcp.json, or .cursor/mcp.json in a project — from its
docs, not run here.) The model loads on the first decide (--preload loads it at launch);
calls run one at a time and protocol messages are answered as they arrive. Measured 2026-09-23
on the Mac (M4 Max, MiniCPM5 2B, two model conversions running alongside): over stdio,
initialize is answered in 15 ms and tools/list in 0.2 ms; four questions on a 119-token
state take 348 ms once the model is loaded (4.2 s on the first call, 3.7 s of it the load);
from a Claude Code 2.1 session a three-question classification round-tripped in 1,064 ms
including the load, 285 ms of it decisions, and from Codex 0.154 in 4,131 ms with the load.
On a quieter run of the same server earlier that day the Claude Code round trip was 766 ms.
Both revisions of the protocol are served — the initialize handshake of 2025-11-25 and
earlier (what Claude Code and Codex send today) and the per-request _meta form of
2026-07-28 (server/discover) — by SystemOneMCPServer and SystemOneMCP in the kit;
runMCP in CLI/main.swift is six lines over them.
# GUI (iPhone or Mac)
xcodegen generate
open Decide.xcodeproj
# headless (macOS), the same function the GUI calls
swift run -c release decide-cli ask --state "The order arrived with the box crushed." \
--noul "Does the customer want a replacement?" \
--choice "What is it about?|billing|damaged delivery|how-to" \
--score "How urgent?|can wait|this week|today"
swift run -c release decide-cli bench --model minicpm5-2b # shared vs direct prefill
swift run -c release decide-cli --list-models
# one decision per line of stdin — a semantic grep
gh issue list --json title -q '.[].title' | swift run -c release decide-cli filter \
--noul "Is this a crash report?"
# a decision model's own fixture: token ids, answer slots and probabilities, row by row
swift run -c release decide-cli parity --model decider-0.8b \
--fixture fixtures-decider-0.8b.json --states states.json
swift run -c release decide-cli parity --model openthai-systemone \
--fixture fixtures-openthai-systemone.json # JSON states render themselves; --bundle <dir> reads an unpublished port
# questions about an image: decider-2b-vision (3.3 GB on first use), every question read in one pass
swift run -c release decide-cli ask --image frame.png --grid 256 \
--state "You control the right paddle. The image shows the current game screen." \
--choice "What should you do right now?|move paddle up|move paddle down|stay" \
--noul "Is the ball moving toward your paddle?"
# its fixture (the zoo's models/decider-2b-vision): the author's ids, slots and fp32 probabilities, run by run
swift run -c release decide-cli parity --fixture fixtures-decider-2b-vision.json --images <the fixture PNGs> \
[--reference <the zoo's Swift run .json>] # --timed <n> timed passes after the checked one
Hands-off, for a recording or a smoke run: Decide.app/Contents/MacOS/Decide -autoplay
sorter -model minicpm5-2b opens that tab, loads the model and presses the screen’s own
sample button (-trigger <path> waits for that file first; -delay <s> after Ready).
Room check loads its own model, the image one: -autoplay room -rooms <dir> [-grid 256] -log 1
presses Check all once it is ready and writes the run to Documents/room-result-
Apple M4 Max, macOS 27.0 (26A428), Release, sequential engine, MiniCPM5 2B int8
(minicpm5-2b) unless noted; other GPU work was running on the machine at the time, so the
timings are an upper bound, not a quiet-machine figure. Dates are when the numbers were
taken.
Checklist (2026-09-22): the sample lease, 565 tokens, twelve questions. Read once in 316 ms, then twelve answers in 532 ms. All twelve read the lease correctly: no subletting without consent, an early-termination fee, pets allowed, no rent increase during the term, the landlord repairs the dishwasher, rent by bank transfer, a 12-month term, 60 days’ notice, smoking not allowed. (Asked as a yes/no, “is smoking allowed?” came back yes at 0.82; asked as a choice — allowed / not allowed / not mentioned — the same model says not allowed at 1.00. The sample asks it as a choice.)
Sorter (2026-09-22): the sample folder, twelve invented documents, four named folders. Twenty-four decisions in 1,558 ms. All twelve land in the intended folder (eleven at confidence 1.00, one at 0.63); five are listed under Needs you — the invoice and the payment reminder (a payment), the lease renewal, the NDA and the neighbour’s letter (a reply, signature or decision by a date) — and the three manuals, the paid receipt, the warranty, the thank-you card and the note are filed without a flag.
Clipboard, watching (2026-09-22): eight copies, two decisions each, 202–211 ms per copy (269 ms for the first, which pays the engine’s specialization at a new length). All eight kinds are named as intended — an address, a secret key, a phone number, a link, a date, an email address, ordinary prose, a tracking number — and against “a shipping address” the address alone is “paste as-is”, the phone number “part of what you need”, the rest “not what you need”.
Autofill (2026-09-22): the sample email (12 lines, with an old address as a decoy), copied once. Prefilled once, then seven decisions — one per field, plus one for where the address ends — in 1,370 ms; all six fields filled, the new address chosen over the old one (street 0.64, its last line against the three lines after it).
Speech gate, sample (2026-09-22): eight utterances, median 64 ms per decision. Five pass: the four requests and “Hold on, let me find my keys.” (0.89); the three remarks are held back at 0.01–0.17.
Search, sample (2026-09-22): six passages in 487 ms, median 65 ms each; the returns passage ranks first at P(yes) 1.00.
Drive (2026-09-23): the recorded run, 17 seconds hands-off, 52 ticks, 52 decisions, median 33 ms each, 22 rocks passed, no crash; the screen paces itself at 0.3 s per tick so the Mac stays watchable. The shape is what makes it work: asked “which move?” with stay / move left / move right, the same model stayed in its lane into a rock four times in twelve states; asked “which lane?” with each reachable lane described by what lies ahead in it, it never picks a rock lane when a clear one is offered (12/12) and keeps its lane while it is clear. What it cannot do is compare two bad lanes — offered “rock 1 ahead” against “rock 2 ahead” it took the nearer rock and crashed at row 30 of an earlier take — so the road never puts rocks in different lanes on consecutive rows: at most one reachable lane is ever blocked within the two rows the descriptions cover, and a clear option always exists.
Columns (2026-09-23): the sample CSV, twenty support tickets, three columns — topic (six-way choice), what the customer wants (six-way choice), mood (a three-level score) — sixty decisions in 3,930 ms, about 200 ms per row including its prefill. Topic and wants read as intended on 18–19 of 20 rows (a discount-code complaint lands in “none of these” rather than billing; “cancel my subscription” reads as wanting a refund); the mood scale puts the two thank-you notes at happy and the complaints at upset. There is no urgency column: asked how soon a short ticket needs an answer, in any shape tried, the model says “today” for nearly every row.
Command guard (2026-09-23): the sample log, fourteen commands, in 1,166 ms — six run
(status, test, cat, an in-project sed, a new branch, an install), six ask first (a force
push, a delete outside the project, a production DROP TABLE, a production namespace
delete, rm -rf node_modules, sudo rm -rf /), two refused (a download piped into a
shell, a private key posted to a paste site). Nothing the policy refuses or holds got
through; the two conservative verdicts are the node_modules delete (ask, not run) and the
disk wipe (ask, not refuse). With the three options named alone the same model asked about
nine of the fourteen; naming each option with the policy’s own words for it is what gives
the split above.
Context (2026-09-23): the sample transcript, thirteen tool results, 1,832 tokens by the model’s tokenizer → seven kept, 958 tokens, thirteen decisions in 847 ms. The four results the question depends on (the session code, the failing test, the grep, the changelog line) score 0.97–1.23 on the none / some / most scale; the weather, the pull-request list, the README, the release script and the pagination code score 0.00–0.71 and drop; the file listing, the git log and the API doc sit at 0.85–1.04 and stay. The line is 0.8.
Typing (2026-09-23): “just got the job offer!! dinner tonight to celebrate?”, typed by the hands-off run — positive, sharing news, 😀; three decisions in 279 ms at the last pause.
Per decision, shared vs from scratch (decide-cli bench: one state, eight questions; the
first three rows 2026-09-21, a 150-token state and questions of 40–60 tokens, median of 3 runs;
the recurrent hybrids below them 2026-09-24 with nothing else on the GPU, median of 6 runs, the
state 111–139 tokens in each model’s own prompt form):
| Model (catalog id) | ms per decision, state shared | ms per decision, every prompt from scratch | prefill + 8 decisions on one state |
|---|---|---|---|
MiniCPM5 2B int8 (minicpm5-2b) |
64.5 | 108.8 | 541 ms vs 1013 ms |
MiniCPM5 1B int8 (minicpm5-1b) |
24.8 | 48.2 | 273 ms vs 468 ms |
Qwen3 0.6B 4-bit (qwen3-0.6b) |
15.8 | 33.6 | 163 ms vs 308 ms |
decider 0.8B int8 (decider-0.8b) |
186.1 | 824.7 | 3,157 ms vs 11,421 ms |
OpenThai-SystemOne 0.8B int8 (openthai-systemone) |
119.8 | 695.9 | 1,568 ms vs 6,180 ms |
Qwen3.5 2B Decision int8 (qwen3.5-2b-decision) |
220.0 | 1,297.2 | 2,816 ms vs 11,429 ms |
APUS Decision v1 4B int8 (apus-decision-v1-4b) |
1,210.8 | 3,080.8 | 11,400 ms vs 25,741 ms |
System One scorer 4B int8 (system-one-scorer-4b) |
719.2 | 5,850.7 | 6,979 ms vs 43,730 ms |
Shared = the state is prefilled once and each question rewinds the engine to it
(Decision.Timing.reusedTokens = the state’s tokens); from scratch = the whole prompt every
time. A recurrent hybrid (the last five rows) cannot rewind its state, so the kit checkpoints it
after the state’s prefix — the recurrent state copied, 10 MB in under 1 ms on the 0.8B–2B
models — and each question returns there and prefills only its own tokens, one per step on
these decode-only graphs. The answers are bit-identical to the from-scratch path on the
fixtures of the first three (44, 50 and 58 rows) and, for the two 4B models, on eight questions
over two states with one revisited. Before coreai-models 0.2.7-zoo the two columns were the same
on the hybrids: every question replayed its prompt from the first token.
The same command on the iPhone 17 Pro (iOS 27.0, 2026-09-24, the phone at thermal state
“nominal” from start to finish, the app in front, the catalog’s own bundles; decide-cli bench
run inside an app on the phone, kit 0.7.1 with --repeat 6, unplugged unless noted; a figure
marked kit a88ec6a is from that commit, before the checkpoint, with --repeat 3):
| Model (catalog id) | ms per decision, state shared | ms per decision, every prompt from scratch | prefill + 8 decisions on one state |
|---|---|---|---|
MiniCPM5 2B int8 (minicpm5-2b) |
100.8 | 220.8 | 1,007 ms vs 1,983 ms |
laya multilingual (laya-multilingual, 256-token window) |
47.1 | 46.4 | 390 ms vs 458 ms (kit a88ec6a) |
decider 0.8B int8 (decider-0.8b) |
517.4 | 2,463.1 | 8.8 s vs 33.6 s |
OpenThai-SystemOne 0.8B int8 (openthai-systemone) |
263.1 | 1,653.2 | 3.6 s vs 14.7 s |
Qwen3.5 2B Decision int8 (qwen3.5-2b-decision) |
1,040.1 | 6,150 (kit a88ec6a) |
13.9 s vs 54.2 s (the shared runs on the charger; from scratch on kit a88ec6a, because kit 0.7.1’s runs went hot) |
The phone reads MiniCPM5 2B at about 1.6× the Mac’s time per shared decision and laya at 4×.
The three recurrent hybrids pay their S = 1 prefill at the phone’s rate, so a state’s first
question costs seconds there. With the state checkpointed, each later question on it costs
263–1,040 ms; before the checkpoint every question paid the whole prefill (kit a88ec6a:
1,527 ms per shared decision on openthai-systemone, 6,063 ms on qwen3.5-2b-decision).
Agreement with the published bf16 readout of the same models on the same 144 authored rows
(SemIf’s authored144 fixture, its direct rendering byte for byte, its evaluate.py metric):
| Model | Prompt tokens identical | Argmax agreement | max / mean |Δp| | Mean family balanced accuracy (kit) | Published bf16 | Accuracy | Catalog temperature | Brier, raw → catalog | ECE (10 bins), raw → catalog |
|---|---|---|---|---|---|---|---|---|---|
| MiniCPM5 2B int8 | 144/144 | 141/144 | 0.183 / 0.020 | 0.681 | 0.686 | 0.701 | 2.93 | 0.455 → 0.411 | 0.167 → 0.071 |
| MiniCPM5 1B int8 | — | — | — | 0.513 | — | 0.535 | 9.974 | 0.636 → 0.623 | 0.184 → 0.157 |
| Qwen3 0.6B 4-bit | 144/144 | 59/144 | 1.000 / 0.580 | 0.333 | 0.440 | 0.340 | 11.882 | 1.299 → 0.722 | 0.650 → 0.185 |
| decider 0.8B int8, card temperature 1.03 | — (its own prompt form) | — | — | 0.753 | — | 0.771 | none (its card’s 1.03) | 0.296 | 0.061 |
On the iPhone 17 Pro (iOS 27.0, 2026-09-23) the same MiniCPM5 2B int8 bundle, read at the same catalog temperature, answers 143 of the 144 rows as the Mac does — the one difference is a row the Mac ties at 0.500 / 0.500 and the phone reads 0.473 / 0.478 — with max |Δp| 0.0049 against the Mac’s rows (mean 0.0011), 144/144 prompt tokens identical, accuracy 102/144 = 0.708, mean family balanced accuracy 0.688 by the same evaluator (evidence 0.752, rule 0.735, candidate 0.579); 115 ms median per decision with the phone at thermal state “serious”.
Brier is the multi-class score (sum over options of (p − 1[correct])², SemIf’s definition), ECE
the top-label expected calibration error over 10 equal-width bins, both from decide-cli
calibrate (2026-09-23, the same Mac); conformance/calibration.py gives the same numbers from
the rows it writes (--out, --out-raw). A chat model’s decisions are read at the temperature
its catalog entry records (CatalogEntry.calibration): fitted, least NLL, on SemIf’s 108
perturbation rows — the same situations with the options reversed, the criterion reworded or
irrelevant context added — and reported on these 144. No answer moves (0 of 144 argmaxes); the
probabilities get less extreme. The two sets share situations: every perturbation row was made
from one of 36 rows here, and 36 of their 72 states are the same text. Fitted instead on two
task families of these rows and read on the third, which shares nothing with them, minicpm5-2b
takes 1.69, 2.79 and 2.52, and the 144 held-out rows go from ECE 0.167 to 0.067 and Brier 0.455
to 0.424 — per family 0.279 → 0.174 (candidate selection), 0.185 → 0.169 (rule application) and
0.096 → 0.123 (evidence interpretation, already close raw). Fitted on these 144 rows themselves
the value would be 2.34 (ECE 0.078). The perturbations break minicpm5-1b (accuracy 0.435 on
them, 0.25 on evidence interpretation), so its temperature is high and its probabilities near
flat; fitted on two families here and read on the third it would be 2.23–2.65, and the held-out
ECE 0.040 against the record’s 0.157. No temperature makes qwen3-0.6b better than a flat guess
over three options (NLL 1.176 at the top of the grid, 11.882, against ln 3 = 1.099 for the flat
guess). The temperatures were fitted on three-option choices; a yes/no and a score are read at
the same one until they are fitted apart. decider-0.8b keeps its card’s 1.03: fitted on the
perturbation rows it would be 1.16, which reads no better here (ECE 0.061 → 0.062, Brier 0.296 →
0.299). TypedDecisions.Configuration.temperature sets one for every question type; decide-cli
calibrate --fit <rows> --report <rows> --record <file> fits one on an app’s own labelled rows
and writes the record.
The screens above were measured at temperature 1, before the kit read minicpm5-2b at 2.93.
Their choices are the same at either temperature; their probabilities were more extreme. What
depends on a probability beyond its winner was re-read at 2.93 on the screens’ own samples
(2026-09-23, the Mac; the raw answers re-read, which is exact for one softmax): Autofill still
fills every field of the sample email — the address line, the lowest, at 0.36 against its 0.3
floor (0.64 raw) — Context keeps the same 7 of 13 tool results at its 0.8 line (the nearest,
read docs/api.md, at 0.84 against 0.85 raw), and Typing’s sample line stays positive
(expected level 1.86 against 2.00 raw). Search’s order in its score mode has not been
re-measured.
So minicpm5-2b is the default: its int8 decisions track the fp32 model. The official 4-bit
Qwen3 0.6B bundle does not — it answers the last option far too often — so use it for its
speed only where a wrong decision is cheap.
A model trained for decisions. decider-0.8b (catalog kind decision, Decision.Format.decider)
was fine-tuned to answer typed questions at an answer slot; the kit renders its own prompt
form and reads it at its card’s temperature. On the model’s 44-row fixture (decide-cli
parity, 2026-09-23, int8 bundle): all 44 rows, the 255-option one included, are
token-identical, slot-identical and argmax-identical to the author’s fp32 readout, max |Δp|
0.0088, mean 0.0010. The 255-option row is 1,965 tokens and takes 11–14 s. It ships as a
decode-only graph on a recurrent hybrid, so a prompt is prefilled one token at a time. Before
the checkpoint every row was prefilled whole: median 343 ms per fixture question, about 490 ms
per clipboard kind and 710 ms per sorter need — five to seven times minicpm5-2b’s
shared-prefix figures (2026-09-23). With the state’s prefix checkpointed, a later question on
the same state prefills only its own tokens: 186 ms per decision against 825 ms from scratch in
the table above. On the two demo questions it was not more accurate
than the chat model (kinds 5/8, needs 11/12). Its place is where the probability must mean
what the model’s own API would report; the screens default to minicpm5-2b.
A decision model with a head of its own. openthai-systemone (catalog kind decision,
Decision.Format.slot; iApp’s OpenThai-SystemOne, Thai + English, Qwen3.5-0.8B tower,
Apache-2.0) replaces the LM head with a 256-way slot head read at a <|ts_answer|> control
token: option i is slot i, the last slot means “none of these” (Decision.Answer.abstain), and
a choice may list up to 255 options (TypedDecisions.maxOptions). The bundle declares the
head and its temperature per question type in its
metadata.json; the kit renders the author’s control-token layout, one question per row. On
the author’s 50-row fixture (decide-cli parity, 2026-09-23, Thai, English and JSON states,
rows of 2–16, 40 and 255 options): tokens, slots and argmax identical on 50/50 for both the
int8 bundle (max |Δp| 0.0226 on one two-option row, mean 0.0009) and the fp16 reference
(0.0051 / 0.0003). On SemIf’s 144 English authored rows, its own form, the same evaluator as
the table above: mean family balanced accuracy 0.725 (int8). Speed on the Mac, each row
prefilled from its first token (2026-09-23): a three-question Thai ticket 351, 316 and 429 ms
for its 57-, 63- and 83-token rows; over the fixture’s 50 rows 354 ms median per question; the
255-option row 7.2 s. Like decider-0.8b it is a decode-only graph on a recurrent hybrid, one
token per step; with the state’s prefix checkpointed, a later question on the same state
prefills only its own tokens — 120 ms per decision against 696 ms from scratch in the table
above. Two things to know: its author’s API lays several questions in one
sequence and answers them in one pass, and those answers can differ from the one-question
rows the kit sends (up to 0.375 on the fixture’s requests) — the kit’s rows equal the
author’s single-question API exactly; and Thai is cut the way the reference tokenizer cuts it
(at every combining mark), which the Swift tokenizer alone does not do — SlotPrompt.Encoder
does the cutting, and 23 of the 50 rows differed before it did. iPhone 17 Pro (iOS 27.0, the
same int8 bundle, 2026-09-23, thermal state “serious”): tokens, slots and argmax identical on
50/50 there too, max |Δp| 0.0210 and |Δabstain| 0.0213 against the fp32 readout (the Mac’s
0.0226 / 0.0222), the 255-option row included; on SemIf’s 144 rows the phone scores the Mac’s
109/144 and 0.725 exactly. 1.9 s median per question over the fixture’s 50 rows (the Mac’s 354
ms), 2.1 s median per SemIf decision, the 255-option row 41 s (7.2 s).
A decision model for browser actions and workflow steps. apus-decision-v1-4b (catalog
kind decision, Decision.Format.sharedState; APUS AI Lab’s APUS-OpenJev-v1-4B, a Qwen3.5-4B
fine-tune, English + Chinese, Apache-2.0) keeps its LM head and answers at the letters A–P
after one user turn under its chat template: Shared state:, the state, a JSON task whose
criteria carry the letters, Answer:. Its own primitives are a choice among 2–16 described
criteria and a yes/no on a proposition; the kit renders a score as a choice over its levels,
and its author applies no calibration. On the author’s 48-row fixture (decide-cli parity,
2026-09-23, int8 bundle) the kit’s prompts are token-identical to the author’s compiled ones on
all 40 choice and yes/no rows, argmax 40/40, max |Δp| 0.0055, mean 0.0002; the 8 rows of the
author’s third primitive (score_level, a yes/no on one proposition under another name) are
outside the kit’s kinds. On SemIf’s 144 English rows, each rendered as a choice, the same
evaluator as the table above: mean family balanced accuracy 0.906 (MiniCPM5 2B 0.681 and
Qwen3.5-4B zero-shot 0.821 on the same rows). The price is the prompt: a 4B on a decode-only
graph reads about 15 ms per token on the Mac. Prefilled whole, a two-question workflow request
took 2,069 and 1,540 ms for its 134- and 107-token rows and a SemIf row 1.96 s median
(2026-09-23); with the state’s prefix checkpointed, a later question on the same state pays
only its own tokens — 1,211 ms per decision against 3,081 ms from scratch in the table above.
Mac only until an iPhone number exists (the bundle is 5.8 GB).
A calibrated decision model in plain text. qwen3.5-2b-decision (catalog kind decision,
Decision.Format.decisionFunction; chaoliangUNSW’s Jev-Style-Qwen3.5-2B-Decision, a
Qwen3.5-2B-Base fine-tune, English, Apache-2.0) keeps its LM head and answers at the letters
` A, B, … (a leading space) after a plain-text prompt with no chat template: a fixed header,
[State], [Question], [Options] as A. … lines, Answer:. Its calibration temperature is
folded into the published weights, so the readout is a softmax at T = 1; its author reports ECE
0.017 on 1,500 held-out examples and 82.3 % on five decision tasks (the author's numbers, not
re-measured here). A yes/no is a choice over yes / no, a score a choice over its levels, up
to 26 options. On the author's 58-row fixture (decide-cli parity, 2026-09-23; 34 choice rows
with 2–16 options, 12 yes/no, 12 score with 2–10 levels) the kit's rows are token- and
slot-identical to the author's and argmax-identical to the converted fp32 reference on 58/58,
for both the int8 bundle (max |Δp| 0.0079, mean 0.0017) and the fp16 reference (0.0058 / 0.0008).
On SemIf's 144 English rows, each rendered in the author's form, the same evaluator as the table
above: mean family balanced accuracy 0.798 (int8), 872 ms median per decision. The author's own
card example, a chipmaker news sentence over four sections, comes back Business 0.684 /
Science/Technology 0.309 / World 0.005 / Sports 0.002 through the kit; the card quotes 0.70 /
0.29 / 0.005 / 0.002 from MLX bf16. Speed on the Mac, each row prefilled whole (2026-09-23): a
three-question request on that sentence (a choice, a yes/no, a five-level score) 817, 633 and
766 ms for its 82-, 73- and 88-token rows; over the fixture 986 ms median per question, the two
1,500–2,000-token rows up to 13.6 s — a recurrent hybrid on a decode-only graph, about 10 ms per
token. With the state's prefix checkpointed, a later question on the same state prefills only
its own tokens: 220 ms per decision against 1,297 ms from scratch in the table above. The
published MLX checkpoint is not in the Hugging Face layout (18
convolution kernels transposed, 61 RMSNorm scales stored without their +1); the zoo's converter
puts them back, proved against the author's MLX bf16 readout (58/58, max |Δp| 0.012), and the
bundle is that converted checkpoint. 3.0 GB int8, the size of qwen3.5-2b`, so it ships to
iPhone too. iPhone 17 Pro (iOS 27.0, the same int8 bundle, 2026-09-23, the phone at thermal
state “serious” throughout): the author’s 58 rows are token- and slot-identical on the phone and
argmax-identical to the fp32 reference on 58/58, max |Δp| 0.0070 (the Mac’s 0.0079), the two
1,700-token rows included; the sentence above with a choice, a yes/no and a five-level score comes
back Business 0.729 / Science/Technology 0.254 / World 0.012 / Sports 0.005 on the phone and
0.731 / 0.253 / 0.012 / 0.005 on the Mac for the same three rows (the card example’s own row
is two tokens longer, hence its 0.684). The price is the same S = 1 prefill at the phone’s rate,
each row prefilled whole (the checkpoint is not measured there yet):
2.7, 6.0 and 7.4 s for those 80-, 71- and 82-token rows, 11.3 s median per fixture question (the
Mac’s 986 ms), the 1,743-token row 155 s — a question a second on the Mac is a question every ten
on the phone, so on the phone this model is for a decision that can wait.
A scoring head instead of letters, under a non-commercial license. system-one-scorer-4b
(catalog kind decision, Decision.Format.scalar; pngwn’s system-one-qwen3.5-4b-scorer, a
Qwen3.5-4B-Base LoRA with a scalar scoring head, English, CC BY-NC 4.0 — its ticket training data
is non-commercial and the model inherits that; CatalogEntry.license carries it and systemone
models shows it) has no letters to read: every option is its own row — State:, the state,
Question:, Option: — read by the head at the row’s last token, and the rows’ numbers are
softmaxed together at the author’s calibration temperature (1.75, fitted on a validation split;
the bundle declares it). The author’s encode keeps the question and the option whole and cuts
the state from its end so a row fits 384 tokens; the kit does the same. The author reports 0.707
accuracy and ECE 0.044 on a 576-question test split, 0.135 for the raw head (the author’s
numbers, not re-measured here). On the author’s 48-question, 280-row fixture (decide-cli
parity, 2026-09-23; 28 choice questions with 2–32 options, 10 yes/no, 10 score with 2–10
levels, six states cut to fit) the kit’s rows are token- and slot-identical on 280/280 and the
question argmax matches the author’s fp32 readout on 48/48, for both the int8 bundle (max |Δp|
0.0110, mean 0.0012) and the fp16 reference (0.0037 / 0.0004). On SemIf’s 144 English rows, each
option a row, the same evaluator as the table above: 121/144, mean family balanced accuracy
0.844 (int8), 2.3 s median per three-option decision with the GPU shared with a conversion run.
A support ticket with three questions — which team, is the customer angry, how urgent on four
levels — took 1,862, 1,094 and 2,198 ms for its 3, 2 and 4 rows (123, 80 and 163 tokens in
all) with every row prefilled from its first token (2026-09-23). A choice costs one row per
option; with the state’s prefix checkpointed, each row prefills only its question and option —
719 ms per decision against 5,851 ms from scratch in the table above. Mac only (5.1 GB int8);
no iPhone number.
An encoder instead of a language model. laya-multilingual (Decision.Format.encoder;
convaiinnovations’ laya, the multilingual checkpoint — an mmBERT-base encoder with a typed decision
head, Apache-2.0) reads a whole question in one forward pass: [CLS] <type> question: <instructions>
[SEP] [MASK] option [MASK] option … [SEP] <state> [SEP], one logit per position, each option read
at its marker and softmaxed within the question. Nothing is generated and there is no KV cache: what
the questions on one state share is its tokens (prefill tokenizes it once), and a recently asked
question’s tokens are reused as well. The bundle declares itself (decision.head == "encoder") with
its window, head budget, special ids and the calibration it ships (a temperature per question type
and option count, fitted on 4,415 examples by the LiteRT port, choice 6–10 kept at T = 1). On the
publisher’s 201 multilingual fixture rows (decide-cli parity, 2026-09-23, the fp16-weight wfp16
bundles, the fixture’s own temperature, T = 1):
| Window | Compute | Tokens / markers identical | Argmax (choice + score) | max |Δp| | Rows within 1e-3 |
|---|---|---|---|---|---|
| 256 | CPU | 201/201 | 81/81 | 9e-6 | 201/201 |
| 512 | CPU | 201/201 | 81/81 | 9e-6 | 201/201 |
| 256 | GPU | 201/201 | 81/81 | 5e-6 | 201/201 |
| 512 | GPU | 201/201 | 81/81 | 5e-6 | 201/201 |
| 256 | Neural Engine preference, three runs | 201/201 | 79–80 | 0.15–0.61 | 168–183 |
On SemIf’s 144 English rows, at the shipped calibration, the kit reproduces the official model’s
readout: 144/144 argmax, max |Δp| 2.9e-6, mean family balanced accuracy 0.611 (evidence 0.619, rule
0.634, candidate 0.581), accuracy 0.590, macro-F1 0.592 — the official model’s own 0.6114 / 0.5903 /
0.5917 on the same rows (0.590 is this model’s accuracy, not its balanced accuracy). Speed on the Mac
GPU (decide-cli bench, the 109-token sample state and eight questions, warm, median of 3, nothing
else on the GPU): 11.5 ms per decision at the 256 window and 18.9 ms at 512 with the state shared,
12.1 / 19.4 ms from scratch; the state and its eight decisions 100 / 160 ms shared, 185 / 240 ms from
scratch, which re-tokenizes the state and the question every time. Tokenizing a question’s head and
options costs 2.3 ms (median) here; kept, it takes a decision’s wall clock from 14.2 to 11.7 ms over
the 201 rows at 256. The bundle loads in about 1.1 s, most of it the tokenizer, whose 256k-token
vocabulary is about 150 MB resident (145–162 MB over three loads); the process is 480–500 MB after its
first decision. The shipped bundles run on the GPU: the Neural Engine takes only an fp16-compute
graph, which misses the answer bar, and asking for the Neural Engine with the shipped graph (the last
row) gives answers outside the bar that change from run to run. A structured instructions value is
written with its non-ASCII text as is, where the publisher’s client escapes it, so such a question
can tokenize differently from the publisher’s; plain-text instructions are unaffected.
laya-multilingual on the iPhone 17 Pro (iOS 27.0, the catalog’s ios/wfp16-s256 bundle,
the same code, 2026-09-23, through a headless harness that runs the fixture rows on the phone): the
201 rows are token- and marker-identical there too, argmax 81/81, max |Δp| 8e-6 on the GPU and
9e-6 CPU-only, every row within 1e-3 of the official model. One decision costs 47 ms with the
state shared (decide-cli bench, the table above), 53–55 ms median per fixture row on the GPU
with the row’s tokens given (p90 57–60 ms; the same at thermal state “fair” and “nominal”), 69–70
ms through TypedDecisions with the state’s tokens kept and the question tokenized on the phone,
73 ms CPU-only — about four times the Mac GPU’s 11.5 ms, where MiniCPM5 2B takes about 1.6× its
Mac time. The bundle loads in 1.2–2.5 s and the process peaks at 357 MB; the zoo’s ahead-of-time
ios-h18p compile of the same graph gives the same 201 rows at 51 ms. A Neural Engine
preference misses the bar on the phone as on the Mac: 196/201 rows within 1e-3, argmax 80/81,
max |Δp| 0.33, at 82 ms, with the process at 1.4 GB (that run and the CPU-only run at thermal
state “serious”).
Questions about an image. decider-2b-vision (catalog kind visionDecision; Mapika’s
decider transplanted into Qwen3.5-2B’s vision-language model, Apache-2.0) answers the same three
shapes about an image and a state, every question of a call read in one pass at its own answer slot
(KitVisionDecider, CoreAI.decide(image:…); ask --image above). It is not a TypedDecisions
and not behind serve: the wire form has no image field. On the zoo’s fixture (decide-cli
parity, the model downloaded from the catalog pin, M4 Max, macOS 27.0, Release, 2026-09-29; the
author’s fp32 read-out as the reference):
| Arm | Runs | Slots | ids and slots = author | Argmax = author | Full-vocab top-1 | max |Δp| | Mean of run means | Decision, median ms |
|---|---|---|---|---|---|---|---|---|
g256 |
35 | 49 | 35/35 | 49/49 | 49/49 | 0.0103 | 0.00027 | 440 |
g448 |
35 | 49 | 35/35 | 49/49 | 49/49 | 0.0080 | 0.00026 | 797 |
| text only | 6 | 10 | 6/6 | 10/10 | 10/10 | 0.0027 | 0.00024 | 346 |
The letter logits are bit-equal to the zoo’s own Swift run of the same files on 108/108 slots, and
a second pass repeats them bit for bit. The times run from the image file to the answers (143 / 275 /
76 tokens at the median) with another job holding the GPU at 80–90 %, so they are an upper bound: the
zoo measured 322 / 584 / 284 ms on this Mac with the GPU idle, from its AOT asset. The tower is 29 ms (g256) / 71 ms
(g448) of it; the rest is the decoder. The first load downloaded 3.3 GB (the decoder and the g256
tower; the g448 tower, 663 MB, when first asked for).
Room check (2026-09-29, the Release app hands-off; six FLUX.2 klein 4B renders made for it,
chosen by eye in file-name order before the model saw them). One press checks the six rooms in
7.5 s at g448, 1.24 s per room at the median (1.22–1.31 s from reading the PNG to the answers:
358 prompt tokens, the tower 0.06–0.07 s, the decoder 1.13–1.20 s). Two rooms come back READY,
three RECLEAN and one CHECK, its lamp read as off at 0.77. At g256 the press takes 5.7 s, 0.95 s
per room, and that room is READY. Of the 30 g448 answers, 29 match what each render was prompted
to show. The miss is folded white towels on an unmade bed, read as “no” at 0.86 — above the line,
so no person is asked. The zoo’s CLI, reading the same rooms from its AOT asset, gives the same 30
answers at both grids (max |Δp| 0.0005). On the 30 rooms the questions were first measured on, the
app matches the CLI’s earlier run 150/150 at both grids (max |Δp| 0.0018). Loading takes 1.7–1.9 s
and the warm-up decision on a blank image 1.1–1.4 s, both before READY; the first launch after a
build took 21 s for the two. The GPU read 0–6 % before each run and the machine’s GPU lock was held
through it, but other sessions share this Mac.
Room check on an iPhone 18 Pro (iOS 27.0 (24A437), the same Release app hands-off with the model
sideloaded into its store, on USB power, 2026-09-29). One press checks the six rooms in 20.2 s at
g448, 3.37 s per room at the median (3.33–3.39 s: the same 358 tokens, the tower 0.25–0.27 s, the
decoder 3.04–3.11 s). That is 2.7 times the Mac’s 1.24 s. Two more g448 presses took 21.6 s (the
first launch after install) and 21.2 s. All 30 answers match the Mac’s (max |Δp| 0.0009), and so do
the verdicts: two READY, three RECLEAN, one CHECK with its lamp at 0.77. At g256 two presses took
15.0 and 17.0 s, 2.47 and 2.79 s per room (226 tokens), with the same verdicts and answers as the
Mac’s g256 run (max |Δp| 0.0004). The thermal state read nominal before and after every run.
Loading takes 7.5–8.4 s and the warm-up decision 3.5–4.7 s, both before READY. The first launch
after install took 10.1 s and 5.2 s, but it read graphs that an earlier app with the same bundle id
had compiled on the phone, so no cold specialization was measured. The app was signed as
com.daisukemajima.decidervisiongate, the one team profile on the Mac that listed this phone.
What the shape does to a small model’s answer. Every question above was tried in
several shapes before it went in (the CLI’s filter is how). With MiniCPM5 2B, a yes/no on
a short text leans yes: “is this what the purpose needs?” says yes to a phone number, a
URL and an API key (P(yes) ≥ 0.7), “does this document need something from you?” says yes to
a manual, “is the speaker asking the assistant to do something?” says yes to three of four
remarks. The same model, asked to choose — “which one is this text?” with an article per
option, “what does this document need from the reader?” with “nothing: a receipt, a manual,
terms to keep” as the first option, “a request for the assistant” against “small talk, a
remark, or talking to someone else” — separates them; and “is this text a shipping address?”
on a no / partly / completely scale puts the address alone at completely. Prefer a choice
with a named “none” option, or a scale, over a bare yes/no when the state is short. The
yes/no shape still reads well on a long state (the checklist’s lease).
iPhone 17 Pro (iOS 27.0, airplane mode, the same app and the same model, 2026-09-23):
the model loads in 22 s the first time after install and 1.4–4.1 s once the system has it
cached; the lease reads in 508 ms and its twelve answers take 1,134 ms (the first run after
install: 1,740 / 1,589 ms); the email’s seven decisions take 2,001 ms; the folder’s
twenty-four take 2,579 ms. The clips in coreai-assets kit/decide/*-iphone.mp4 show these
runs; the first-run figures were read back over devicectl from the app’s -log 1 file.
iPhone 17 Pro, the five newer screens (iOS 27.0, airplane mode, the same app and the
same int8 bundle, the recorded runs, 2026-09-23): Drive, 17 seconds hands-off: 53 ticks, 53
decisions, median 64 ms each, 19 rocks passed, no crash. Columns: the twenty tickets × three
columns, 60 decisions in 7,420 ms, every cell the same as on the Mac (20/20). Command guard:
fourteen commands in 1,906 ms, the same 6 / 6 / 2 verdicts at the same confidences. Context:
the thirteen tool results in 1,454 ms, the same seven kept, scores within 0.01 of the Mac’s.
Typing: three decisions in 484 ms at the last pause. The model loads in 1.5–5.5 s from the
cache; the hands-off -log 1 runs before the recording gave the same figures within a few
percent (Drive 80 ticks in 25 s at a 63 ms median). The same bundle on both machines gives
the same decisions; the phone takes about twice the Mac’s time per decision. Clips:
coreai-assets kit/decide/{drive,columns,guard,context,typing}-iphone.mp4.
The benchmark’s own harness (fstandhartinger/jevbench
2fa63fa, v1.4.0, typesafe adapter) sent the 231 public items to decide-cli serve at coreai-kit
adbc755, one question per request, on the M4 Max; accuracy per tier and the hard tier’s ECE are
JevBench’s own scoring (argmax of the returned probabilities). Latency is per request (p50 and p95
over all 231, the maximum over the hard tier), taken on a shared GPU and before the checkpoint: at
that commit a recurrent hybrid prefilled the state again for every question.
| model | easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max | bundle on disk | note |
|---|---|---|---|---|---|---|---|---|---|
apus-decision-v1-4b |
1.000 | 0.986 | 0.559 | 0.347 | 5.94 s | 142.09 s | 217.6 s | 5.4G | two other model servers on the GPU⁴ |
decider-0.8b |
1.000 | 0.833 | 0.414 | 0.320 | 1.69 s | 29.10 s | 90.6 s | 1.2G | server alone |
laya-multilingual |
0.896 | 0.403 | 0.342 | 0.273 | 0.02 s | 0.20 s | 0.3 s | 649M | 256-token window¹; server alone |
minicpm5-2b |
0.979 | 0.708 | 0.459 | 0.241 | 0.14 s | 1.19 s | 1.8 s | 2.5G | zero-shot chat model³; server alone |
openthai-systemone |
1.000 | 0.819 | 0.324 | 0.449 | 0.62 s | 25.93 s | 39.0 s | 1.0G | server alone |
qwen3.5-2b |
1.000 | 0.750 | 0.441 | 0.245 | 2.74 s | 50.22 s | 62.3 s | 2.8G | zero-shot chat model³; server alone |
qwen3.5-2b-decision |
1.000 | 0.778 | 0.405 | 0.235 | 3.24 s | 92.89 s | 138.5 s | 2.8G | two other model servers on the GPU⁴ |
system-one-scorer-4b |
1.000 | 0.861 | 0.505 | 0.164 | 8.96 s | 61.61 s | 72.0 s | 4.7G | 384-token rows²; two other model servers on the GPU⁴ |
¹ The bundle (macos/wfp16-s256) reads a 256-token window: 95 of the 111 hard items were
truncated to it (hard states run to 3,677 tokens); no easy or standard item was (the longest is
107 tokens).
² The kit follows the author’s encode, which cuts the state from its end so that each row (state +
question + option) fits 384 tokens; on 70 of the 111 hard items the state was cut.
³ A chat model asked zero-shot under the kit’s JSON decision prompt: minicpm5-2b at its catalog
calibration temperature 2.93 (fit on SemIf perturbations108), qwen3.5-2b with no calibration
(temperature 1).
⁴ Three model servers ran on the same GPU at once. A solo re-check of 30 items per model returned
bit-identical probabilities, in about half the time.
Apple’s on-device foundation model on the same 231 items, through decide-cli serve --backend fm
at this branch (FoundationModelDecisions), the same harness and driver, one question per request,
on the same Mac later the same day (other lanes’ builds on the CPU, no other model server):
| model | easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max | bundle on disk | note |
|---|---|---|---|---|---|---|---|---|---|
apple-foundation-model, session shared per state |
0.875 | 0.722 | 0.387 | 0.602⁵ | 0.36 s | 2.04 s | 3.4 s | 0 | system model, guided generation⁶; 4 × 422⁷ |
apple-foundation-model, new session per question |
0.875 | 0.722 | 0.387 | 0.602⁵ | 0.34 s | 2.04 s | 3.4 s | 0 | system model, guided generation⁶; 4 × 422⁷ |
⁵ Not a calibration figure: the framework exposes no logits, so every answer’s probability is 1 and
ECE is 1 − accuracy. The calibration axis does not apply to this backend.
⁶ Each question is one guided generation under the answer’s schema — a choice an enumeration of its
option ids, a noul a Bool, a score an Int within its levels — sampled greedily, the state in the
session’s instructions. Shared and fresh answered all 231 items identically; with one question per
request they send the same prompt. On eight questions about one state (decide-cli bench --backend
fm), a fresh session per question is the cheaper of the two: 340 ms per decision against 366 ms on
the shared transcript, which the framework serves from its cache but which grows with every answer.
⁷ The framework’s guardrails or the model refused four items as unsafe or sensitive (one names a
credit card’s last four digits); a refusal is a 422, which JevBench counts as wrong. No prompt
overflowed the 8,192-token context window (the longest hard item is 3,999 input tokens). The
yes side of a noul is over-chosen: 53 yes against 35 expected over the 74 yes/no items.
JevBench’s published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed
ones included, and are not comparable to these accuracies. Every request and answer is in
mlboydaisuke/coreai-decision-models-public231.
The iPhone 17 Pro’s results on the same items, for five of these models, are in
docs/SYSTEM_ONE.md.
Sources/QuickStart.swift — the take-home: one typed function, no UI. The GUI and the CLI
both call it; decide(image:state:questions:) beside it is the same for a question about an
image (decider-2b-vision, ask --image).CLI/main.swift — argument shell over that function, plus bench, oracle, parity
and filter (the numbers above), serve, a shell over the kit’s SystemOneServer
(Sources/CoreAIKit/Decide/SystemOneServer.swift), and mcp, one over
SystemOneMCPServer; clients/ — a curl and a Python request to it.Sources/DecideRuntime.swift — the one loaded TypedDecisions the screens share.Sources/Form*.swift (Autofill; FormPage.html is the checkout page it fills through one
JavaScript call), Checklist*.swift, Sorter*.swift, Search*.swift, SpeechGate*.swift,
Drive*.swift, Columns*.swift, Guard*.swift, Context*.swift, Typing*.swift — the
ten screens; Intents.swift — the Shortcuts actions; hooks/claude-code-guard.sh — the
guard as a Claude Code hook;
DocumentText.swift — a file as text (plain, Markdown, PDF, image via Vision);
Autoplay.swift — the hands-off runner.github.com/john-rocky/coreai-kit → product CoreAIOps (or CoreAIKit for
TypedDecisions alone)NSMicrophoneUsageDescription and NSSpeechRecognitionUsageDescription only
for the speech gate; the other screens need nothing (the file importers are system pickers)com.apple.developer.kernel.increased-memory-limit for the 2.7 GB
MiniCPM5 2B bundle, and for decider-2b-vision: without it the decoder’s first specialization
dies with std::bad_alloc on an iPhone 18 Pro (the model zoo’s gate)