Hard-won, verified notes on Apple’s Core AI (iOS/macOS 27) — what the docs don’t spell out.
Published as a page per topic at john-rocky.github.io/coreai-model-zoo,
so each note has a URL that can be cited. Apple documents the API surface; these notes cover the
part that is not written down anywhere — what the runtime does when you run it, the thresholds,
and the failure modes. Plain HTML, so a reader without JS, a search crawler and a coding agent
all get the same words.
llms.txt indexes every one of them.
For the long-form version of the same material, read
The Art of Core AI — a free book.
undocumented-answers.md — the questions Apple’s docs leave open,
with measured answers: the AOT threshold, the iOS-only dynamic-KV miscompile at seq ≥2048, whether
a 4B fits on the ANE, what the chunk threshold really dials, and why a hand-written kernel forces
a single-token export.coreai-error-index.md — got an error string? search it here: every
exact Core AI / coreai-torch / coreai-build / Swift-engine error this project has hit, verbatim
as a heading (cannot unwrap empty odiec_module_t, Failed to acquire the source buffer for the
ViewOp, not a valid substitution for source shape 1, failedToSpecialize,
NSPOSIXErrorDomain Code=2, …), each with when it appears, the verified cause or “Not
isolated”, the fix, the log or Apple issue behind it, and the OS / toolchain. coreai doctor
links its findings here by anchor.coreai-overview.md — what Core AI is, the 3 Apple repos, the .aimodel
format, the PyTorch → .aimodel → Swift-runtime pipeline.conversion-guide.md — how do I convert a PyTorch model to .aimodel?
The short answer, the decision table (zoo recipe / Apple preset / re-author / the five lines /
ONNX), the canonical TorchConverter API, and the gotchas that cost real time.ship-playbook.md — the end-to-end runbook: converted .aimodel →
CoreAIKit Swift engine → app → on-device (AOT + sideload + headless self-test for RTF) → publish
(HF + zoo + post). The stage checklist + cross-cutting traps (gate-before-port, JIT→AOT, tokenizer
retag), validated shipping Parakeet in one session.card-definition-line.md — the one sentence every own card and
repo opens with (what Core AI is, the export tools, one measured number with source and date),
why that number, and how to update it with tools/card_first_line.py.compute-units-and-authoring.md — ANE vs GPU vs CPU: the
static/BC1S/Conv2d/per-head/fp16 ANE rules vs the dynamic/fused/custom-kernel GPU rules, the
macOS↔iOS export split, and the PSNR verification gates. Read this to choose a target.performance-ceiling.md — reality check: where Core AI LLM decode tops out
(Mac GPU near its ceiling, MLX gap structural, fusion closed), what AOT does/doesn’t do, and why ANE is an
energy play not a speed one. Read before chasing a “dramatic speed” win.pipelined-engine.md — read this first for decode speed: riding Apple’s
coreai-pipelined engine = 3.5× over a hand-rolled per-token loop with ZERO custom kernels
(qwen3.5: Mac 204 tok/s, iPhone 50.3–51.5). The decode-only loop-free export, the extra-states
engine patch, the chunk=1 / warmup-256 traps, LUT-vs-linear int8, oracle gating, and what
fits/doesn’t (Gemma 4’s PLE doesn’t — yet).int8-head-and-decode-measurement.md — the int8 LM head
decode lever (untie + absmax int8, ~half the per-token read) and how to measure the win honestly:
controlled-bench vs in-app, VLM image_embeds dilution (+48% text core → +36% VLM), thermal-robust
ratios, and the SwiftUI O(n²) re-decode that masquerades as a slow model.fm-provider.md — zoo models behind Apple’s LanguageModelSession
(WWDC 339): CoreAILanguageModel(resourcesAt:) = the whole integration (verified, incl.
hybrid/SSM bundles on the patched pipelined engine), plus the own-conformance recipe that
adds tool calling (verified round trip) and the protocol traps (dead prewarm, no guided
generation on pipelined, re-prefill tax).custom-metal-kernels.md — TorchMetalKernel (WWDC 325): the API, the
register-then-add order, MSL embedded in the .aimodel, GPU-only, what to (and not to) kernelize.
Still the tool when a model CAN’T ride the pipelined engine (e.g. Gemma 4).rwkv7-recurrent-linear-attention-coreai.md — the
no-KV playbook for pure-recurrent / linear-attention LLMs (RWKV-7, Mamba-2, gated-delta-net):
the matrix-state decode recurrence lowers to standard ops (no custom kernel); O(1) fixed-size
states, no KV cache wired via SSMState + fused end-of-step writes; the pipelined engine can’t
drive a no-KV model (positional keyCache/valueCache) → a custom backend binding states BY NAME
(like BitVLA); the recurrence-protecting quant recipe (int8keepproj); teacher-forced gating.
Shipped RWKV-7 1.5B, iPhone 17 Pro 25.2 tok/s.tensorops-quantized-kernels.md — the layer UNDER custom kernels
(WWDC 330): TensorOps quantized matmul (int4/int8 = OS 26; fp4/fp8/int2 + E8M0 scale planes = OS 27),
cooperative tensors, the FlashAttention recipe, and the M5/A19 GPU neural accelerator — the
compute-bound/prefill lever hand-rolled MSL can’t reach.
prefix-cache-kv-reuse.md — cross-turn KV reuse: turn-2 TTFT
0.23 s vs 23.3 s = 101× at a 4k context, proven lossless (greedy token-identical). The
orthogonal speed lever nobody’s shipping on-device yet.spec-decode-design.md — speculative decoding on the pipelined decode
bundles: the only lever past the decode bandwidth wall (verify K tokens per forward); design +
feasibility. GDN-hybrid (Qwen3.5/3.6) static-S verify companion:
spec-decode-hybrid-verify-design.md.spec-decode-ngram-dense.md — the dense case, measured:
training-free n-gram drafting at 1.34–1.96× lossless on a shipped 30B bundle, no
re-export and no drafter. Verify cost is a staircase (S≤3 free, S=4–8 ~1.47×, S≥9 ~2.3×),
so K belongs at the top of a plateau — the wrong K turns the same drafter into a slowdown.tensorops-zoo-impact-and-kernel-wins.md — applied
survey: where TensorOps quantized kernels and pure custom-Metal wins actually pay across the zoo.apple-models-bench.md — measured numbers for Apple’s own
coreai-models export recipes — the README Apple didn’t write (21 recipes, zero official
numbers). §2026-09-15: the June ios/ IRs of qwen3-0.6b / qwen3-4b AOT-compiled for the Neural
Engine on the GA toolchain (31/31 regions, no re-export) and handed to the on-device token-match gate;
results live on the models/qwen3-{0.6b,4b}-official/ cards.coreai-vs-mlx-speed.md — every measured Core AI–vs–MLX decode
comparison (same M4 Max, same protocol) + the causal decomposition of the gap: where Core AI
wins, where it structurally can’t, and why.aot-and-specialization.md — specialization, AIModelCache /
AIModel.specialize(), and AOT compile (xcrun coreai-build compile → .aimodelc,
--preferred-compute neural-engine). The first-run-latency mitigation path.jit-distribution.md — one .aimodel for every iPhone generation: the 18 Pro
specializes shipped graphs up to the 1.6 GB whisper itself (4.4 s first load, 0.3 s after, the app’s footprint
untouched), an AOT bundle of another architecture is refused — also when the JIT files sit beside it — and
the layout rule that follows (ios/ = JIT IR, ios-<arch>/ = AOT for the LLM class only).compression-reference.md — coreai-opt quantization & palettization
API reference (int4/int8, granularity, mixed-precision, joint); the LM-head/embedding lever.ane-silicon-reference.md — the ANE silicon map, distilled from the
reverse-engineering guide arXiv:2606.22283: which compression forms stream vs fold per generation
(palettization is the only always-streaming form), the M(n)=H(n+12) family/capability staircase,
hard caps (~128 programs/process, 2→4.72 MB working set, >4094 slice-inf on A13/A14), the measured
decode-vs-encoder verdict with its bandwidth mechanism, and the warmup/idle/power rules a fair ANE
bench must follow. External direct-route numbers on M-series — not our Core AI-route measurements.coreai-beta-mpsgraph-kvwrite-bug.md — EXC_BREAKPOINT
(SIGTRAP) at the first execute: the data-indexed in-graph KV write SIGSEGV (FB23024751 / apple#5),
platform-agnostic (GPU too), host-cache workaround.qwen3.5-static-ane.md — can a GDN hybrid (Qwen3.5) reach the Neural
Engine through Apple’s static iOS path? No, on the release toolchain: the stock exporter has no
iOS builder for it, and the community q=1 builder AOT-compiles to 0 ANE regions in every arm —
the compiler names the fp32 gated-delta recurrence (Incompatible element type for ANE), 6 layers
fail like 24, and removing the recurrence exposes a second, unnamed blocker. Control: Apple’s dense
Qwen3-0.6B with uncompressed fp16 weights lands 19/19. What a port would need, and the cheapest
next instrument (fp16 recurrence under the Mac GPU oracle gate).coreai-ane-partition-cost.md — an op the ANE cannot run (topk
is the usual one) charges a fixed cost, not one that scales with its work: cutting k 30× buys
nothing. Count boundary crossings, not ops. Self-contained reproducer (apple/coreai-torch#66).ane-region-count-and-fp32-ops.md — zero ANE regions is a
failure mode, not a tie: Granite-Embedding-97M at fp32 forms 0 regions under
--preferred-compute neural-engine (the silent GPU fallback), fp16 forms 13, and removing two fp32
ops (the softmax’s dtype=torch.float32, the pooling head’s .float()) forms 1. Count
*ANE_region_*.mlir.bc, because *ANE_region* also matches each region’s directory; IOReport
shows placement where xctrace’s ANE intervals read 0; --compute does not override an AOT
bundle’s placement; w8 / w6 / w4 negatives for this graph. Base M4 Mac mini, contributed (PR #36).coreai-zero-sized-dim-abort.md — Pass failed:
MPSCommonRuntimeCanonicalization / MPSNDArray … buffer is not large enough: a 0-length split
section or a width-0 output converts fine and then aborts the process on GPU and ANE (CPU runs it).
Arrives from generic postprocess code that sizes a section by subtraction. Also: pin the compute
unit explicitly when isolating — SpecializationOptions.default() can fall back to CPU and a
fallback reads as a pass (apple/coreai-torch#68).porting-text-embedders.md — an embedder is not a small LLM:
gate the ranking, not the cosine; k-means skips nn.Embedding, which is 24% of a big-vocab
embedder; fp16 tables are free when the checkpoint is bf16; int4 fails on the interval. Plus
the three ways a retrieval benchmark lies, all of which make the numbers look better.granite-embedding-97m-port.md — ModernBERT to Core AI
(Granite-Embedding-97M): the local window is inclusive radius 64 = 129 keys, and a radius of
63 reproduces the final embedding to cos 0.99995 — only a per-layer gate (1e-4) catches it;
two RoPE thetas by layer kind, no attention norm on layer 0, the config’s absolute position type
is not what runs. The tokenizer is the contract (PAD 179935, body S−2 then CLS/SEP, no stripping,
ignore_merges=true; a gate whose corpus cannot separate a mutation is not gating it). w8 palettes
shrink the bundle 22% and buy no speed because the fp32 table is 71% of the bytes; the fp16-table
lever is exact on a bf16 checkpoint and still unshipped. fp16 whole-model fails the layer gate.decider-0.8b-port.md — a System One decision model on Core AI
(Mapika/decider-0.8b): the gate is probability parity, not a transcript — the author’s fp32
letter readout vs the bundle on 44 rows (int8hu max |Δp| 0.0084, argmax 44/44; the fp16 floor
is 0.0050 because the graph emits float16 logits). The Swift pipelined engine has no logits
(sequential = 2 states only), so probabilities come from an AOT .aimodelc through the Python
runtime — whose GPU JIT is wrong even at 0.8B on 26A428 (MTL4 error 1, zero logits) and
which leaks one IOSurface per call (~25k S=1 calls per process). Also SemIf’s authored144 read
the same way on two shipped bundles (MiniCPM5-2B 0.681, Qwen3.5-4B 0.821).decider-systemone-op-design.md — design of
CoreAI.systemOne(state:questions:options:) for coreai-kit: wire shape, the prompt-builder port
line by line, tokenizer parity on the 44 fixture rows, and the readout primitive (recommended: a
completion-synchronized read-last-logits call on the pipelined engine; fallback: the zoo’s
N-state low-level runner). Not implemented.decider-2b-vision-port.md — an image-input decision model on Core AI
(Mapika/decider-2b-vision): the overlay’s Qwen3.5 tower at a fixed grid plus a new ids-input Qwen3.5
hybrid decoder that takes the image rows as a static input and derives M-RoPE in the graph
(image_rc lets one decoder take any grid up to 256 tokens). Positions are gated by exact plane
equality, because a 1-D position error moves this fixture’s p by ≤ 0.0012; the fp16 tower fails (worst
row cosine 0.528) and fp16 storage with fp32 math passes; Pillow’s resize pass order is worth up to 17
levels; the int8 body error sits in layers 0–11 and three fp16 layers {0, 2, 5} fix it, confirmed on 500
held-out photo runs; coreai-opt’s symmetric_with_clipping does not clip; the prefill function
grows the AOT compile by 3.8 GB but not the .aimodel; Swift’s JIT of the .aimodel reads correctly.clef-flash-port.md — a hidden-state decision model and its head on Core AI
(Cloudflare/clef-flash): the 9B decoder returns final-norm hidden rows at every position (no vocabulary head)
and the author’s joint schema head runs as its own graph, with the untied lm_head as a host gather table; the
fixed grid’s price measured with the author’s code alone; fp16 passes while a few hidden rows drift, and fp32
attention does not lift them; the int8 error sits in layers 0–15 and the bisect’s search over up to six fp16
layers found no set meeting its rule (layers 0–11 ship, tested on a held-out set); a dynamic-shape head AOT
built with --expect-frequent-reshapes aborts at its first call, so four static buckets ship; the plain
attention chain compiled for the GPU is wrong from 4,032 keys and key blocks of ≤ 2,048 fix it; the Swift host
copies Pillow’s integer resampler and Python’s JSON rendering; Swift’s JIT of the fp16 .aimodel equals the AOT
asset bit for bit.kev-port.md — a LoRA + pointer-head decision model on Core AI, at 0.8B and 4B
(jaredpalmer/kev-0.8b, kev-4b): the author’s merge script, checked by bit-equal logits; the delimiter ids come from
the base tokenizer, not the kev-4b repository’s Qwen3-era files; the overlay’s text decoder returns hidden rows, and
the scan form sets a call’s cost: both sizes ship 128 tokens a call with the overlay’s fp32 Metal kernel for the
Gated DeltaNet recurrence (a row holds 3,968 tokens), the in-graph chunk breaks at S = 32 already in fp32, and a dynamic
query length was measured and not shipped (its footprint grows while the call length keeps changing, on the Mac and
the iPhone; an AOT cache entry is named by the function’s type, so caps collide); int8 measured and not shipped (no set met
the bisect rules, and a bisect’s rows do not transfer between int8 variants); --expect-frequent-reshapes adds an
fp16 copy of every linear and a compiled graph holds int8 linears as fp16; the host’s head in float64 with Python
3.12’s sum() and Python’s str(); the shared prefix is exact on a static graph; Swift’s JIT of the .aimodel
equals the AOT asset at both sizes; Kev-0.8B on the iPhone 18 Pro at the default memory limit, and Kev-4B’s iPhone AOT asset crashing at
load; four of ten invented fixture names found in use on the web.d1-port.md — an option-token decision model with pictures on Core AI
(LiquidAI/d1-3B): the vocabulary head replaced by the tied embedding rows of 2,134 option ids on the host (the
provider’s log-sum-exp cancels in a softmax over the options); the processor’s resize is torch’s uint8 bicubic
kernel, not Pillow’s; the vision tower’s fp16 math misses a row and fp16w32 ships, while a toy hid fp16w32’s real size;
int8 on the MLP linears alone passes (every MLP and conv projection fails, int4 is a cliff); an AOT cache entry is
named by the kind of linear and S, not the bit width; dynamic positions without --expect-frequent-reshapes
re-specialize per position length (on the phone a call right after it can be wrong), a static form specializes once
but costs more per call; three Neural Engine walls; on the iPhone 18 Pro neither decoder form loads without the
increased-memory-limit entitlement, and the phone’s JIT beats the Mac-compiled AOT asset per call; on the Mac an int8
.aimodel’s JIT is slower than its AOT asset and gives other bits while the fp16 one matches it bit for bit, so the
two platforms ship different decoders; the export writes local paths into main.mlirb, and the stripped copy (one
program object, or nothing is stripped) gives the same rows.openthai-systemone-port.md — a slot-head decision model on Core AI
(iapp/OpenThai-SystemOne): the LM head replaced by a 256-way head read at a control token —
model.* weights, a resized embedding, vocab_size 256 as the logits width; temperatures from the
checkpoint tensor, not the author’s card; the author’s one-pass API differs from independent rows
(max |Δp| 0.375); and the Swift tokenizer’s grapheme-cluster cut of Thai, fixed in the kit.laya-multilingual-port.md — an encoder-type decision model on Core AI
(convaiinnovations/laya, multilingual): one forward pass per question, options read at mask markers; the
publisher’s own package as the oracle (frozen fixture reproduced bit for bit); a massive [CLS]
activation (~1.4e4) from layer 11 puts the official model’s own two attention paths 4.8e-2 apart, so
the per-layer gate is two-tier (1e-4 absolute, 2e-4 relative); fp16 compute misses the bar while fp16
storage with fp32 compute is exact (645 MB); the Mac GPU computes at fp32 precision, the Neural Engine
preference gives non-deterministic answers on the fp32-compute graph; the Swift host’s two Unicode traps.julia-1-port.md — the laya decision head on mmBERT-small (SupersonicLabs/Julia-1):
the host is where it differs (options are the criteria text as given, strict encoding refuses instead of
cutting, T = 1, no act head); the publisher’s CPU result reproduced three ways; the layer bar re-measured
(the publisher’s own SDPA-vs-eager distance is 2.9e-4 at the final norm, so laya’s 2e-4 does not carry over)
and the residual traced to the final LayerNorm of a different torch build; fp16 storage is not free on an
F32 checkpoint (|Δp| 0.023); the Neural Engine does not take the fp32 graph.d1-omni-port.md — an encoder decision model with an image and an audio tower on Core AI
(LiquidAI/d1-omni-600M): three graphs (the decision graph at seven static lengths from 64 to 4,096, SigLIP2
vision per crop, FastConformer audio per clip bucket) with the masks as float inputs and the media prefix
written by the host; the publisher’s fp32 floor sets a 2e-5 module bar; L = 4,096 attends in two blocks of 2,048
keys; fp16 passes at every length while int8 fails and compiles to fp16-sized weights, and the Neural Engine
misses the bar; torchvision’s uint8 resize, the position-table resize and the mel copied in NumPy and Swift bit
for bit, with a libjpeg-turbo-exact baseline JPEG decoder (ImageIO is up to 3 levels off); the MLIR debug
locations carry the exporting machine’s path until strip_debug_info; Swift’s JIT of the .aimodel equals the
AOT asset; swift-transformers’ tokenizer takes 0.4 s on a 3.4k-token state; a question costs 7.35 ms at L64 and
16.2 ms at L256 on the M4 Max GPU, so the host picks the smallest bucket that holds the row.apus-decision-v1-4b-port.md — a letter-readout decision model as an HF-id swap
(apus-ailab/APUS-OpenJev-v1-4B): the unchanged Qwen3.5-4B exporter; the author’s openjet_runtime as the fp32
oracle (transformers 5.16.1 pinned); the Shared state: + JSON task turn under the chat template read at
A–P; the 16-layer early-exit path left out; the fixture’s short states and absent near ties, said plainly.qwen3.5-2b-decision-port.md — inverse MLX Qwen3.5 conv/RMSNorm layout, two-oracle proof, plain-text space-prefixed letter readout, Mac GPU gates and supervisor-supplied coreai-kit measurements.system-one-scorer-4b-port.md — pngwn/system-one-qwen3.5-4b-scorer (cc-by-nc-4.0): pinned LoRA merge, scalar-head export, distinct input-embedding/tokenizer/output widths, author truncation/readout, Mac GPU scalar gates and the limits of one-wide engine checks.coreai-torch-042-lowering-changes.md — which of the
nine semantic lowering changes in coreai-torch 0.4.2 can reach a shipped bundle, decided by
converting the same minimal module under both versions, diffing the graph, and running the ones
that differ. Exactly one moves values (integer true-divide: 7/3 was 2.0); fp16 batch norm
rewrites the graph and changes nothing, because the runtime picks the kernel for a composite.
Graph-identical proves output-identical; graph-different proves nothing until you run it.spotlight-rag-third-party.md — running Apple’s WWDC26
SpotlightSearchTool (local RAG as one Tool) behind a third-party zoo model via
KitLanguageModel: only .toolCalling is needed (not guided generation), the tool returns
metadata-not-body (hydrate with a companion fetch_note tool), guidance level is a token gate,
and the thinking-model /no_think mitigation. Verified example: coreai-kit/Examples/SpotlightChat.dynamic-profiles-local-models.md — WWDC26 DynamicProfile
(242) routing between two local zoo models (0.6B triage ↔ 4B expert) in one
LanguageModelSession, fully on-device/airplane-mode — the config Apple’s on-device↔PCC demo
doesn’t show. The body-purity rule, switch re-prefill cost, two-resident-model footprint, and
why the model-decision channel must be guided-gen (not a tool) on the stock engine. Example:
agent-demos/DualProfileChat.visual-intelligence-third-party-model.md — running
YOUR own model (CLIP / RF-DETR) behind the system Visual Intelligence camera/screenshot search
(WWDC26 297): IntentValueQuery + SemanticContentDescriptor, model-agnostic by construction (no
model param, no capability, no entitlement), and the real gate — running a model in the query’s
background-launch memory budget. Example: coreai-kit/Examples/VisualIntel.agentic-security-checklist.md — pre-ship checklist for
on-device LLM agent apps (WWDC 347+343): indirect prompt injection, the Lethal Trifecta,
.onToolCall/.historyTransform guardrails, App-Intents risk-based confirmation +
authenticationPolicy + OwnershipProvidingEntity.evaluations-framework.md — Apple’s Evaluations framework
(WWDC 298/299/335) mapped to this project’s oracle/margin gates; the disallowed-trajectory
injection test; a Vault-style on-device eval suite.compression.md — this project’s LLM-specific empirical compression notes
(int8 floor, per-subsystem sensitivity); pairs with compression-reference.md.stateful-kv-cache.md — stateful decode export, dual/hybrid KV state,
the sliding-window ring buffer, the dynamic prefill+decode graph.swift-runtime.md — the Core AI Swift API, driving .aimodel from Swift,
non-standard architectures, macOS/Xcode 27 setup (incl. running a second Xcode outside /Applications without sudo).muse-glimmer-port.md — Meta’s 30B VLM text tower (NoPE full layers,
gated attention, weight-less Q/K norm). Two traps that generalize: _mutate_state_dict never
runs on the shared slice (nested text tower + untied head loads to garbage), and
hidden_states[-1] is the final norm, not the last layer. Also the EDGE record for a port
taken on knowing it is Mac-only against an already-converted MLX build.bitcpm-ternary-1.58bit.md — 1.58-bit ternary MiniCPM4-8B: the
zoo’s first sub-int8 packed-GEMM Metal kernel; an 8B running in ~2.1 GB on the iPhone GPU.bonsai-ternary-hadamard.md — Bonsai 2 27B (ternary g128 +
blockwise Hadamard, Qwen3.8-27B hybrid): PQ2_0/PTQ1_0 bit layouts, where the rotation sits,
what stays non-ternary, and the diff against the zoo’s qwen3_5.py. Written before the port.bitvla-1.58bit-vla.md — 1.58-bit Vision-Language-Action (robotics):
image + instruction → 7-DoF actions, fully on-device.s1-mini-port.md — S1-mini (Superwhisper), a Qwen3-0.6B ASR text
normalizer, and the port’s real finding: a passing conversion gate says nothing about a
task model’s task. int4 scored the same 16/16 token-exact oracle as int8 and still
corrupted digits. Also: why a tied head makes *hu the wrong idea rather than a smaller
one, and why an upstream card’s example outputs are not fixtures until you check they
reproduce on the released weights (here: 9/14).ane-vs-gpu-iphone-2026-09.md — Neural Engine vs GPU on the iPhone 17 Pro, one
variable at a time (2026-09): the same-day interleaved protocol (one app, both bundles, A-B-A-B), weight-byte-
matched pairs at 1B and 2B, a third family on Apple’s own builder (Qwen3-1.7B — whose iOS preset is 6-bit, not fp16),
and the 10-minute sustained curves with thermal state per trial: the ANE 8-bit 1B drops from 76 to ~52 tok/s once the
phone reports serious (~70 s in, wired + screen on), so first-minute numbers are not sustained numbers. Battery-%
needs the cable out; the instrument for it is in the note.
§9 (2026-09-16): Apple’s 6-bit / mixed 4-8 / 4-bit g8 recipes at 2B on the ANE — 6-bit runs (1.70 GB of weights,
38.5 tok/s, cold program build 23 min) and now ships as the 2B’s ios-ane-h18p/: GSM8K 173/200 vs 172 for fp32, the
replaced 4-bit bundle 131 — the token gate alone was not a quality gate, and a Mac simulation of a recipe predicts the phone;
4-bit g8 will not compile for the ANE; the specialization cache can be pre-seeded per container.
§10 (2026-09-18): LFM2.5-1.2B, a conv hybrid, on Apple’s static path — the conv history lives inside the KV rows
(position-indexed, read through mask-derived one-hots; the [W,L]@[L,2] selector matmul is the one op the ANE compiler
refuses); 31/31 regions, 38 tok/s, GSM8K at the checkpoint’s level, but the phone’s logits carry O(1) noise the Mac fp16
twin does not (recipe, precision and eps excluded one by one) — not shipped, mixed-fp16 arms prepared.ane-quality-gate.md — the quality gate for a Neural Engine bundle (rule since 2026-09-17):
recipe simulated on the Mac (coreai-opt’s k-means on the fp32 weights + 200 GSM8K) → token gate on the phone → task gate on
the phone; token match alone let a −20-point 4-bit bundle ship. Tooling in apps/AneGate/.minicpm5-1b.md — the clean-LlamaForCausalLM recipe done end-to-end (hybrid
Think/No-Think, untied head, 128K) — the most reusable conversion template in the zoo; re-run
unchanged for MiniCPM5-2B (2026-09), plus why a device-gate prompt is chosen by fp32
margin, not habit. §2026-09-15: the Neural Engine lane — Apple’s stock static iOS export gated on
the phone (teacher-forced sweep + free-run vs fp32 through the unmodified StaticShapeEngine):
the 2B passes at 4-bit palettized, the 1B fails at 4-bit (fp16 embeddings do not rescue it) and ships at 8-bit;
the phone’s 28 GB specialization-cache trap and the zero-byte-stub fix.youtu-mla-port.md — dense DeepSeek-style MLA at 2B on iPhone: latent-KV
attention with an absorbed flash-decode kernel.diffusion-llms-dllm.md — masked-diffusion LLMs (LLaDA): parallel
canvas denoising, bidirectional attention, no KV cache — and how that maps to Core AI graphs.gliner25-decide.md — GLiNER2.5-Decide, zero-shot classification: only the classification
path as one DeBERTa-v3 graph (labels as [L] positions, S = 256 / 512); coreai-torch 0.4.1 divides int / int as
integers, which silently breaks DeBERTa’s relative-position buckets beyond 128 tokens (bake the table); the iPhone
18 Pro is h19p, not h18p; the oracle’s self-consistency and an outside check; three things a bit-exact Swift host
needed; measure the phone after a rest.gliner2-pii.md — DeBERTa-v3 (disentangled attention) NER / schema-driven
zero-shot extraction; the on-device PII-redaction model.glm-ocr-port.md / mineru-port.md /
unlimited-ocr-rswa-static-decode.md — the document-OCR
trio: a Glm4v variant, whole-page parsing on stock Qwen2-VL, and an R-SWA MoE on the stock
runtime with static-shape stateful decode.ovisocr2-port.md — in progress, nothing published; carries its own
handoff (§Start here: one authoring variant left, plus artifact/device/gate state). OvisOCR2 tops
OmniDocBench v1.6 (96.58) over the zoo’s own MinerU2.5-Pro (95.69) and is config-identical to
Qwen3.5-0.8B, so it is a weights swap on shipped authoring. Carries the port’s two transferable
findings: the first non-square vision grid through qwen3_5_vision.py (gated exact in fp32,
a 4-row fp16 tail that is not the blank margin), and the stop token living in
tokenizer_config.json, not config.json — on the config value the bundle never terminates.video-world-models-vjepa2.md — V-JEPA 2: a self-supervised
video world model as on-device action classification (16-frame clips).gemma4-mixedbit-qat-transplant.md — extracting Google’s
mobile mixed-bit QAT weights and transplanting them into Core AI bundles.gemma4-ple-static-input-fm-stack.md — Gemma 4’s
per-layer-embedding table as a static graph input, loaded behind FoundationModels.gemma4-apple-authoring-designs.md — Gemma 4 in Apple’s own
authoring branches (dual bounded/growing KV cache, ring + blocked flash on the ANE) vs the zoo port, and the
converter traps they encode.timesfm-port.md — TimesFM 2.5: the zoo’s first time-series forecasting
foundation model (stateless graph + host RevIN DSP).timestral-port.md — Timestral-209M (Mistral AI): a time-series patch transformer on the
GPU and the Neural Engine. A finite additive mask stops masking when attention scores reach 2.4e5; fp16 compute
overflows through the scores (fp32 scores fix it, and fp16 compute runs 2.4× faster than fp32 on the iPhone 18 Pro’s
GPU, no faster on the M4 Max’s); the Neural Engine’s near miss was its sigmoid / SiLU (written through tanh, the
graph passes, and has the lowest latency on the iPhone 18 Pro in every bucket); R = 2 compiles for the Neural Engine
only with 1×1-conv projections; an AOT compile’s cache entry name ignores the weights; int8’s cliff is the patch
embedding and the head.esam3-port.md — EfficientSAM3: a dropped port (device-verified but redundant
vs the official SAM 3) — kept for what transferred.magenta-rt2-port.md — Magenta RealTime 2: a parked port (not shipped,
not dead — real-time on iPhone was proven against a Mac-only model, on both runtimes, and the
first Core AI blocker is identified in the note). Kept for the ANE sampler rewrite — argmax+gather in a 12-step depth loop costs 12
partition crossings and 34 ms/frame; onehot = (x >= max(x)) then matmul is the same tensor for
free — plus the codec rules (batch the call, keep the iSTFT on the host) and where 4-bit stops
working (79% of the params take it; the logits head and embeddings do not).shieldstral-port.md — Shieldstral-1.0-3B: why a model that answers
with one token should not ship as a decoder (bake the tail, keep two rows of the head), how
to port a checkpoint your export venv cannot load — build the oracle in the venv that
supports it natively, then measure the substitute (ministral3 = Mistral + YARN, cos
1.000000) instead of forcing a config — and the two numbers that invert decode-loop intuition:
quantization buys size, not speed, and a verdict costs the grid, not the text — the same
|ΔP| on an M4 Max and an iPhone 17 Pro, to four decimals.north-micro-vision-port.md — Cohere’s cohere_compass: how to
notice in a minute that a new checkpoint’s vision tower is one you already have (load it
strict=False and print missing/unexpected), the four decoder details that run fine when done
wrong (parallel block, mean-subtracting LayerNorm, a quarter of the layers with no
positional encoding, logit_scale), and the iOS load wall measured 2.39 GiB further out than
this repo believed.lfm2.5-vl-port.md — LFM2.5-VL (450M + 3B): SigLIP2 NaFlex (a Linear patch embedding
over host-flattened patches, a 16×16 position grid resized per image), what baking a square grid
costs a NaFlex model, the antialiased resize a GPU “bilinear” is not — and why judging a
compressed VLM needs an fp16 baseline, because greedy decoding turns any near-tie into a
different tail — and how the same family flips verdict on int4 between two sizes (450M 0/9,
3B 7/9), plus the two per-checkpoint host details (resample filter, and a tokenizer that
does not prepend BOS) that read as a broken port.lfm2.5-2.6b-port.md — LFM2.5-2.6B, where the port was a config change and
the work was surviving a transformers-v5-era checkpoint: a RoPE key that moved, a tokenizer
class that does not exist yet, and a 4.x modeling file that silently ignores a config flag and
would certify a wrong oracle as PASS.qwen3.8-27b-port.md — Qwen3.8-27B, a generation bump that was a
verified weight swap (config + weight map byte-identical to 3.6-27B), and what a
release-day download actually costs: HF CDN at ~0.15 MB/s per connection with a ~2.7 MiB/s
per-IP ceiling, XET stalling live, and the fix — sha256-verified ModelScope through aria2c at
~20 MB/s, seeded into the hub cache by LFS oid. Phase 2 adds the vision path and four
durable lessons: a bf16 full-model oracle is not a valid tower target (gate vs an fp32 tower
ref); the chunked GDN scan’s fp16 doubling-inverse overflows content-dependently at S=32 — it
passed the oracle suite then collapsed on the next real photo, so ship the chunk the dtype
can prove (S=16) and gate chunked-vs-S=1 token equality; multifunction bundles can assert in MPSGraph’s ANE region
pass under JIT (AOT h16c is the python-driver path); capture the oracle’s mRoPE positions with
a hook instead of re-deriving them.minicpm-v-4.7-1b-port.md — MiniCPM-V-4.7-1B: 4.6’s vision code in front of
a Qwen3.5 hybrid with canvas M-RoPE, derived in the graph from the ids and two static shifts (decider-2b-vision’s
contract, the 8×8 grid baked); a last-position logits bar passes 4.6’s 1-D positions, so the planes are gated for
equality. The head is untied yet 4.9e-4 from the embedding; the tower’s LayerNorm inputs overflow fp16 when squared
and the GPU’s fp16 tower still passes; the Mac AOT and the iPhone both store the int8lin tower as fp16, so int8 buys
download, not speed; iPhone 18 Pro device JIT 98.0 tok/s decode; coreai-torch’s debug paths in main.mlirb and how
the strip was re-gated.wevisone-vl-2b-port.md — WeVisOne-VL-2B: built upon Qwen3-VL, with a 9.5M-parameter
pixel decoder that turns the hidden state of each <OBJ> into an alpha matte. The decoder returns last_hidden
as a second output, which the pipelined engine runs but never hands out, so the app drives the four graphs on the
low-level runtime; the pixel decoder as two static graphs (one answer’s queries attend to each other); one fp32 ulp in
a box moves the matte through its 2^k Fourier prior, on the GPU (the log term) and on the host (Darwin’s logf
rounds 7 of the 3,494 possible widths apart from torch’s); the decoder bar D sits on the fp16 floor; the prompt’s
tail padded into one prefill call (30 ms against up to 1.1 s one token at a time on an iPhone 18 Pro); iOS’s
1,024-position KV cap; the image path in the ANE’s static form (host-made M-RoPE; 8-bit passes, no hidden state);
PixelShuffle and ConvTranspose2d on the ANE; why a re-export is never the same bytes.zimage-port.md — Z-Image-Turbo, a 6B Single-Stream DiT text-to-image — and why
it ships Mac-only.glm-image-port.md — GLM-Image: the zoo’s first AR + diffusion hybrid
(9B GLM-4 AR writes visual prior tokens → 7B flow-matching DiT renders).flux2-in-context-editing.md — FLUX.2 [klein] instruction
editing + multi-reference composition with no separate editing model and no ControlNet.flux2-klein-reexport.md — FLUX.2 klein 4B re-exported with Apple’s stock
exporter (apple/coreai-models 97a14be), 15 forms against a per-component fp32 gate: the registry’s
int4 default fails it (the quantized text encoder breaks at the padding the transformer reads), fp16
--single-function ships at 10.88 s per 1024² image on an M4 Max and int8 per-block 32 at 11.60 s;
unquantized multi-function specializes 8 weight copies (62 GB), the Neural Engine compile fails with
exit 0, AOT is slower and the app cannot open it, tiled decode saves 630 MB only with every model loaded.qwenimage21-port.md — Qwen-Image-2.1 (RGBA-Image-2.1), a 7B
block-causal DiT with an RGBA VAE: the DiT runs on the GPU only after an
--expect-frequent-reshapes AOT compile, the text encoder’s <|im_start|> tokens need fp32 compute, and
the 512² image forks inside the model’s own bf16 band.
The Turbo checkpoint (RGBA-Image-2.1-Turbo) reuses the encoder and VAE bundles on a fixed 8-step grid, and its full int8 DiT compiles where the 2-layer probe does not.qwen2.5-omni-audio-understanding.md — Qwen2.5-Omni’s
Thinker as on-device audio understanding (describes sounds, not ASR): de-dynamizing the
Whisper-style encoder (ragged cu_seqlens chunk attention → one batched fixed attention, bit-exact),
TMRoPE collapsing to 1-D, a bit-exact vDSP mel (DFT-as-matmul), audio embeds on a static buffer, and
the two payoffs — AOT clean-mmap weights dodge the iOS jetsam dirty limit (4.5 GB decoder, 5930 MB
free), and a fixed-shape encoder runs on the ANE (0.99 cos → byte-identical text) where the
dynamic decoder can’t.whisper-asr-fixed-decode.md — Whisper large-v3-turbo as on-device
ASR (the zoo’s first official speech-to-text): why the official recipe’s single-step [1,1]
decoder graph can’t transcribe, why a dynamic-length re-export recompiles every step (~15 s/token),
and the fix — a fixed 128-token decoder window (pad, read logits at the real last position;
causal attention ignores the padding; constant shape compiles once → 0.18 s/token, token-exact).
Plus the Swift log-mel frontend (n_fft 400 = DFT-as-matmul) and single-main→GPU routing.funasr-nano-port.md — Fun-ASR-Nano-2512 (SAN-M speech encoder + adaptor +
a fine-tuned Qwen3-0.6B; zh / dialects / en / ja, hotword prompt): the decoder is not stock
Qwen3-0.6B and its position-0 massive activation (125k) overflows float16 — fixed exactly by running the
residual stream at 1/4 with RMSNorm eps × 1/16; the SAN-M encoder’s 1.39× fp16 headroom → float16
weights with float32 arithmetic; keep the tied head tied (int8lin, not int8hu); two LayerNorm epsilons;
N = ceil(L/8) audio rows; no audio marker token; funasr’s dither is a constructor argument; and the
knife-edge accounting that makes 150/155 vs 155/155 the same verdict.embeddinggemma-2-transcribe-port.md — EmbeddingGemma 2’s audio encoder
(frozen) + a reader trained here that reads each token as the closest of 200,000 words, as English speech-to-text:
half_mask is the only mask a padded window needs; 30 s windows with 96 tokens of left overlap, each dropping its
last token; float16 needs a max-abs RMSNorm, and the subsampling LayerNorms must stay plain for the Neural Engine;
near-silent tokens are the float16 failure point on every backend (one float16 step on 12 inputs moves a token from
0.972 to 0.99996) → judge the app path by its transcripts; the Mac’s Neural Engine passes only as a two-bundle cut,
the iPhone’s runs none of the 30 s graphs; the 200,000-word table stays inside the graph; and the runtime traps (an
AOT main.hash ignores the weights, two calls of one function corrupt each other, export debug locations carry the
converting machine’s paths).kokoro-tts.md — Kokoro-82M (StyleTTS2 + iSTFTNet), the zoo’s first text-to-speech:
3 bundles cut at the one data-dependent length + host DSP, the weight_norm-random-init bug
(non-determinism → suspect weight loading), variable length via masked-unrolled bi-LSTM +
frame-masked InstanceNorm + host STFT, the Core AI op rewrites (ConvTranspose1d/iSTFT → zero-insert
adcsr-super-resolution.md — AdcSR, the zoo’s first super-resolution
(one-step diffusion-GAN, pruned SD-2.1 + half VAE decoder): why fp16 NaNs (SD-2.1 attention overflow,
upcast_attention ignored by the SDPA processor) so fp32 ships, the global (not per-tile)
color-match, host-side tiling with an input-size cap, and the CoreGraphics bytesPerRow/no-y-flip traps.sam3-promptable-segmentation.md — SAM 3 as on-device text-prompt
segmentation (the zoo’s first official segmenter): the segmenter bundle format + one-call official
CoreAIImageSegmenter runtime, float16≈float32 fidelity (Δ≤1e-4) verified with the image-segmenter
CLI, redistributing a gated model under the SAM License (ship the LICENSE), and the platform box-origin
trap when compositing masks.depth-anything-3-monocular-depth.md — Depth Anything 3, the
zoo’s first depth model (DINOv2 ViT + DPT head, single static square graph): why the input is
raw [0,1] (ImageNet norm folded in-graph) and the double-normalization trap that faked a
cos-0.9 “engine bug” / “noise” / “non-square bug” for a day; engine cos 1.000000 at any shape;
the squish + resize-back contract and why r≈0.98 vs official is faithful (= the model’s own
504-vs-518 variance); the RoPE-const / cache-free / pos-embed-dtype export patches; fp16 via
.half() not autocast. Sample: scripts/depth_anything_3_sample.py.
voxcpm-tts.md — VoxCPM-0.5B, the zoo’s first diffusion TTS: a family of
graphs (LM + diffusion + VAE + vocoder) resolved as one catalog model.pocket-tts-port.md — pocket-tts (Kyutai), the zoo’s first Kyutai model
and first Mimi conversion: why a fixed-capacity KV cache must be zero-initialised, never
NaN-initialised (0 * NaN = NaN survives a masked SDPA, and a fixed graph cannot slice the
unwritten tail off the way upstream does); preferredComputeUnitKind: .cpu as a partitioning
hazard rather than a compute-unit choice (.cpu and .gpu declare the same allowed set and
differ only in preference — verified on two unrelated models, blast radius from rounding-scale
to anti-correlated); deriving S_MAX from the model’s own generation bound instead of the
fixtures; windowed prefill as a second escape from the large-query SDPA lowering crash; and
gating TTS with an ASR round trip because tensor cosine passes on unintelligible audio.audio8-tts-port.md — Audio8-TTS-Preview-0.6b (DualAR, Fish Audio S2 Pro design: a Qwen2.5-shaped
slow AR + a 4-layer fast AR + a 44.1 kHz DAC-style codec; 11 languages, zero-shot voice cloning): the zoo’s first sampler
inside the graph — the publisher’s top-k / top-p / temperature / Gumbel-max / RAS draw written without a sort (topk +
logsumexp + a cumulative sum, the uniform draws as inputs) so a frame is one runtime call instead of ten (48 → 32 ms on an
M4 Max); where the remaining milliseconds are not (cache size, int8, composite ops, AOT — one variable at a time) and the lever
that is left (the pipelined engine); the 4,097-row head that is exact; the publisher’s bf16 RoPE tables; int8 on the fast AR
as the wrong 53 MB; the codec encoder’s vector quantizers needing fp32 arithmetic; the RAS window quirk; gating TTS with an
ASR round trip and a speaker-verification cosine; the Python bindings’ IOSurface leak.chatterbox-port.md — Chatterbox: the zoo’s first zero-shot voice-cloning
TTS and first multi-network port.music-generation-stable-audio.md — Stable Audio Open:
text→music latent diffusion (T5 cond + DiT + VAE) on-device.sortformer-speaker-diarization.md — streaming speaker
diarization: export only the neural core, port the streaming loop + AOSC speaker cache to the
host.nemotron3-diarization-port.md — Nemotron-3-Diarization, the
8-speaker successor: speaker-cache rules taken from transformers, not from the 4-speaker port; one
left-packed graph for three streaming modes; padding rows zeroed before the Conv1d; a 2-ulp compression
tie that forces float64 scores; and why the Neural Engine error is not LayerNorm overflow.vibevoice-multispeaker-tts.md — VibeVoice, the zoo’s first
multi-speaker / dialogue TTS: dual-LM next-token diffusion, why the speech feedback loop
forces fp16, and why the expectFrequentReshapes hint must be off on iOS.melband-source-separation.md — Mel-Band RoFormer, the zoo’s first
source separation: complex → real arithmetic, and folding STFT/iSTFT into the graph as constant
DFT matmuls so the on-device host needs no FFT.Primary official sources behind these notes: the open repos (coreai-torch, coreai-optimization,
coreai-models incl. its agent skills), the WWDC26 talks 324 / 325 / 326 / 330 (verbatim transcripts in
ondevice/_wwdc{324,325,326,330}_transcript.txt), and developer.apple.com/core-ai/. Verified against
Hugging Face references (convert + numeric parity); on-device iOS 27 notes marked where still in bring-up.