LiquidAI/LFM3-1.7B (LFM Open License v1.0, revision ac2586a34f1a1c03329dbdcdefabd8b6478ddaa7) is the first
LFM3 checkpoint in the zoo. It keeps LFM2.5’s idea — short-conv mixers interleaved with attention, no recurrent
scan — and changes almost everything around it: the conv window, a sliding window on five of the eleven attention
layers, the rotary, an output gate, the norms, and the prompt surface (thinking on by default, XML tool calls).
The decoder module written for it is shared by the three LFM3 multimodal ports in progress (VL, Omni, Audio;
separate pull requests), so this note is also where their language model comes from. Card:
models/lfm3-1.7b.
The engine here is the zoo’s fork of apple/coreai-models
(john-rocky/coreai-models): llm-runner and the zoo’s provider
were built from its branch zoo-0.2 at 661a5d4 (the checkout that ran the gates carried one more local commit, an
unrelated Python module, so its Swift engine is 661a5d4’s), and CoreAIKit pins its tag 0.2.8-zoo. Below, “the
fork” is that branch at 661a5d4 and “the kit’s pin” is the tag.
| LFM2.5-1.2B / 2.6B | LFM3-1.7B | |
|---|---|---|
| 1. conv mixer | depthwise causal conv, kernel 3; state [n, 1, 2048, 2] |
kernel 7; state [13, 1, 2048, 6] (13 conv layers, the last 6 columns of B·x) |
| 2. attention layers | full attention only (6 / 8 layers), 32 q / 8 kv heads, head_dim 64 | 11 = 6 full + 5 sliding (window 256), interleaved with the conv layers; 16 q / 4 kv heads, head_dim 128 |
| 3. rotary | full head_dim, one θ (1e6 / 1e7) | full layers: θ 1e6 on half the head_dim (partial_rotary_factor 0.5, zero-tail, below); sliding layers: θ 1e4 on the full head_dim — two tables per step |
| 4. attention output | o_proj of the heads | output gate: sigmoid(gate_proj(x)) (2048 → 16, one scalar per head, x = the mixer’s normed input) times each head before o_proj |
| 5. norms | pre-norm per block (operator_norm → mixer → + residual → ffn_norm → MLP → + residual), final embedding_norm |
sandwich norm: operator_norm → mixer → post_operator_norm → + residual → ffn_norm → SwiGLU → post_ffn_norm → + residual; the final norm keeps LFM2’s name embedding_norm |
| 6. prompt surface | pythonic calls between <\|tool_call_start\|> / <\|tool_call_end\|>; no reasoning block (1.2B) or <think> opened by the template (2.6B) |
XML calls (<tool_call> / <function=NAME> / <parameter=KEY>); the template opens <think> by default; the vocab still carries LFM2.5’s <\|tool_call_start\|> pair |
Unchanged: hidden 2048, vocab 128 000 with a tied head, a SwiGLU MLP (7680 here), RMSNorm with eps 1e-5,
per-head q/k RMSNorm. The config is transformers-v5 shaped: tokenizer_class: TokenizersBackend (the
tokenizer trap of lfm2.5-2.6b-port.md applies as it is), and rope_parameters one
level deeper than the 2.6B’s — a dict per layer type (full_attention, sliding_attention).
The module is new (coreai_models/models/macos/lfm3.py in the authoring overlay), not a flag on lfm2.py: it
imports LFM2’s conv mixer and _rotate_half, uses the shared primitives (MLP, RMSNorm, SDPA, KVCache),
and writes the attention, the decoder layer and the stack. It was gated against transformers 5.20.2’s fp32 eager
on 17 sequences (8 short questions with thinking off and on, and one 1,648-token document) along three paths — one
call over the whole sequence, S=1 with a 4,096-slot KV, and the engine’s own walk (S=1, KV grown from 256 by
doubling): worst max|Δlogit| 1.22e-3 / 2.61e-3 / 2.61e-3, top-1 2682/2682 on every path (one position whose HF
fp32 top-2 gap is under 1e-3 left out). Three negative controls turn it red: dropping the output gate (23.4 on the
long sequence, top-1 889/1647), rotating every dimension (16.8, 1450/1647), and removing the window (24.3,
1081/1647; the 31-token sequence stays green: under 256 tokens the window never bites).
window_size, with no mask inputThe decode graph keeps the whole KV for all 11 attention layers in one KV pair ([11, 1, 4, L, 128], L
dynamic) and lets the composite apply the window: Lfm3Attention passes window_size=256 to the SDPA
composite, the same path the stock gemma3_text.py takes. The composite’s window branch is the lower-right
causal mask XOR q_idx >= k_idx − (k_len − q_len − W), i.e. “absolute query position − key position < 256”,
which is transformers’ kv_idx > q_idx − sliding_window. The host passes position_ids (the full length) and
the three states; there is no mask input, and the engine’s contract (one KV pair + one fixed extra state)
holds.
Two things were checked before trusting it:
−1e4 where masked). The attention scores’ absmax over
the 17 sequences is 22.8, so nothing leaks through it. A model whose scores reach the thousands would need
a look at this before reusing the path.A ring-buffer KV for the five sliding layers (256 slots) was not built: the decode over a ~3.7–4k KV is only 1.9 % slower than over ~1–1.3k for all 11 attention layers, so a ring on 5 of them would win back about 1 % at a 4,096-token context.
partial_rotary_factor: 0.5 on the full layers does not mean “rotate the first 64 dimensions”. transformers
builds inv_freq over the whole head_dim — its first 32 entries are θ^(−2i/128) and the last 32 are 0 — and
then concatenates the frequencies with themselves and applies rotate_half. The rotated pairs are (i, i+64) for
i < 32; dimensions 32–63 and 96–127 get angle 0 (cos 1, sin 0) and pass through. A port that rotates the first 64
dimensions instead pairs other dimensions at other frequencies; that form was not run here. The control that was
run — the full layers rotating every dimension, with no zero tail — moves the long sequence’s logits by up to 16.8
(top-1 1450/1647). The sliding layers rotate every dimension with θ 1e4.
The module builds both tables once per step from position_ids in fp32 and shares each between the layers of its
type. inv_freq is a plain fp32 attribute, not a registered buffer, so a .to(torch.float16) of the model does
not round it; the sliding θ stays the integer 10000 from the config (the tiny model’s prefill logits match
transformers at max|Δ| 0.0).
The same two GPU-delegate traps as LFM2.5 (models/lfm2.5, its “Two GPU-delegate findings”), and the
same fixes: the conv layers return their new columns and the stack writes all 13 slots of the conv state with
one slice_update per step, and the attention projections keep fp32 weights. For LFM3 the fp32 path is wider:
[1, 1, 16] elementwise work), and the SDPA output is multiplied by it in fp32 before o_proj.The activations themselves fit fp16 with room: over the 17 sequences the largest values were 4.53 (mixer output before its post-norm), 30.2 (MLP output before its post-norm), 55.6 (SwiGLU intermediate), 25.7 (residual stream) — far from 65 504, so the sandwich norm needs no residual scaling.
The fp16 bundle fails the strict parity bar on the long sequence: teacher-forced on the GPU against transformers’
fp32 over the same 17 sequences, max|Δlogit| is 0.353 (p50 0.026), and 6 of the 1,647 counted positions of the
1,648-token sequence flip their top-1, at positions where transformers’ own top-2 gap is 0.003–0.061. It is not
the conversion: the same module run in fp16 on the CPU in PyTorch shows the same width (0.49, 4 flips), the fp16
logits’ rounding alone accounts for 0.008–0.011, and the same graph exported in fp32 and run on the GPU passes at
2.20e-3 with every top-1 right. Read fp16 numbers for this model, and for the VL / Omni / Audio language models
built on the same module, against that floor. The shipped bundle’s int8 weights (int8hu: the MLP and conv-mixer
linears int8 per-block-32, the head untied and int8, the attention fp32), teacher-forced S=1 through main, keep
2637/2682 top-1 = 98.32 % (1027/1035 on the short sequences, 1610/1647 on the long one); the same quantized
weights computed in fp32 in PyTorch keep 2636/2682, so the fp16 activations add almost nothing on top of the
quantization.
The vendor chat_template.jinja (sha256 f45005c5…) uses Python-jinja constructs the Swift renderer does
not handle in every version that ships:
| swift-jinja | vendor template | the bundle’s template |
|---|---|---|
2.3.2 (the pin of apple/coreai-models) |
does not parse: Expected '%}' after for loop.. Got or instead ({% for tool in tools or [] %}) |
30/30 |
| 2.3.6 (CoreAIKit’s pin) | parses; the 10 conversations with tools differ (tojson is a JSONEncoder: sorted keys, no spaces, \uXXXX, \/) |
30/30 |
| 2.5.2 | 30/30 | 30/30 |
30 conversations = 15 scenes with thinking off and on — the 8 questions, a three-turn chat with reasoning in the
history, a turn with a system prompt and no tools, and five tool scenes (a round trip with and without a system
prompt, two calls in one turn, a finished round trip in the history, a tool description full of characters that
JSON escapes) — compared string for string and id for id with transformers 5.20.2’s apply_chat_template. The
bundle carries the vendor template with eight rewrites: parentheses around tools or [], and a _json(v) macro
in place of the template’s tojson calls — it uses tojson when that already prints json.dumps’ format and
writes the format out otherwise. transformers’ jinja2 renders the rewritten template to the same string as the
vendor’s on all 30. The other tokenizer files are the source’s, verbatim.
Watch for one quiet fallback: when the template does not parse, Apple’s CoreAILanguageModel (and the fork’s)
catches the error, logs one CLILogger line (off by default) and encodes the messages’ text joined by newlines
without a chat template; the caller is not told (read in the code, not run). That is why the rewrite is shipped
instead of relying on a newer swift-jinja being resolved.
<|tool_call_start|> /
<|tool_call_end|> tokens, so a probe that recognises LFM2.5 by them picks the pythonic dialect for a model
that calls tools in XML. The zoo’s defaultDialect(probing:) and CoreAIKit’s KitLanguageModel check
for the single tokens <function= and <parameter= first (LFM3Dialect / LFM3ToolDialect).<think>, so
the model’s first token is already reasoning. Stream parsers that start “outside” — Apple’s ThinkTagParser,
and the zoo’s and the kit’s parsers before this port — would send the reasoning into the answer text (checked on
the parsers with synthetic streams). The rule both repos now apply: when the rendered prompt ends with an open
thinking tag (and an optional newline), the parser starts inside it. Apple’s parser is unchanged.</think> after 40–237 tokens (median 56) and answered right; the three runs gave the same text. In all 24 the
answer starts right after </think>, with no whitespace in between. (Run on the main-only build of the same
weights: these 20–29-token prompts never reach prefill, and the shipped bundle’s main gives that build’s
values at all 2,683 positions.)</think> would shift the history. The zoo executor keeps whatever follows </think>
as the answer, where LFM3’s tokenizer_config response_template consumes whitespace there
(</think>\s*), so a reply that began with a newline would re-render a token off transformers’ history. Not
seen in these runs: the prompts of a three-turn thinking-on chat, re-rendered by the zoo provider and by
CoreAIKit’s ChatSession, equal transformers’ render id for id on every turn.LanguageModelSession, with one tool that returns a fresh random value
each run: 3 / 3 through CoreAIKit’s KitLanguageModel and 3 / 3 through the zoo’s ZooLanguageModel. The model
reasons, calls lookup_secret in XML with key: alpha, reasons over the result and answers with the value
verbatim; the reasoning lands in the transcript’s reasoning entries and stays out of the answer in all six runs.maxTokens after the consumer stops at the stop token (the zoo executor drains and records
it, as its type comment explains), so each zoo turn here ran to maximumResponseTokens (2,048), while the kit’s
pin stops at the stop token (50 tokens for the call turn). Left open: in the tool loop, the call turn
re-rendered from its parsed reasoning and call has the generated turn’s token count (50, with no whitespace
after </think>) but not all of its tokens; where they differ was not isolated.main is the S=1 decode graph (the composites’ own causal and window masks) and
prefill is a static 64-token query over the same positions, KV and conv state, with the masks built from the
positions (Lfm3Model.set_explicit_mask()). Exporter flag --prefill-chunk 64 (bundle suffix _pf64, for
“prefill chunk 64”). The weights are shared in the .aimodel (+174 KB for the second graph); the Mac AOT
(--architecture h16c --expect-frequent-reshapes; the architecture codes this port compiled for are h16c on the
M4 Max, h18p on the iPhone 17 Pro and h19p on the iPhone 18 Pro) holds an fp16 copy of the int8 Linears per
function (resources.bin 6.39 → 10.08 GB); when the .aimodel is specialized on load instead, its compile-cache
entry holds a 3.74 GB resources.bin.prefill by name — the fork and the kit’s pin alike: a prompt of N tokens runs
floor((N − 1) / 64) chunks through prefill and the rest one token at a time through main, the last one as
the sampled step (a 1,024-token prompt: 15 chunks, then 64 single steps). Its log line (CLILogger level 1, off
by default in an app): Pipelined engine loaded 'prefill' function: static chunk S=64.main alone: 2637); the
logits at the chunk boundaries equal the main-only path’s at cos ≥ 0.9999922; main gives the main-only
bundle’s values at all 2,683 positions. Greedy, a 215-token prompt that rides 3 chunks gives the fp32 oracle’s
16 tokens (the token gate: conversion/coreai_gate.py, 16 greedy tokens against the fp32 oracle — the PyTorch
module run in fp32); the
stripped .aimodel gives the export’s values at every position, chunked and not..aimodel specialized on load on the Mac), in one measurement window on 2026-10-31 from a
1,024-token prompt: the first token after 0.83 s and a decode of 169.5 tok/s (median of 8); the same .aimodel
compiled ahead of time, on the same engine in the same window, 1.00 s and 105.5 tok/s, so the Mac bundle ships as
the .aimodel. Earlier, timed while other jobs ran on the Mac, a 1,233-token prompt reached the first token after
0.68 s with prefill and 7.4–7.8 s with main only. A tool round trip’s prompts (127 and 197 tokens) ride it
too.main; at a 1,024-token prompt those 64 steps are about half of the
first-token time.--pad-tail, not in a pinned engine yet)prefill as one padded call instead of one token at a time — at a
1,000-token prompt the first token comes after 0.444 s instead of 0.767 s, at 200 tokens after 0.122 s instead
of 0.157 s, with the same decode (M4 Max, 8 runs of each, alternated). It needs an engine that pads the tail:
the one measured is a patched copy of the fork’s engine; neither the fork nor the kit’s pin pads.--pad-tail gives prefill a chunk padded at its end: the padding engine writes token 0 and
position id -1 into the padded rows, and the graph reads the new conv state after the chunk’s real tokens —
columns [n, n + 6) of [state | Bx], n = the number of query position ids ≥ 0 — instead of the last 6. Nothing
else changes: the causal mask comes from indices, so no real query sees a pad; the KV rows the pads write lie
past the cursor and are written again before anything reads them. A full chunk (n = 64) reads the last 6
columns, so an engine that never pads computes exactly what _pf64 computes (all 2,683 positions equal, chunked
and through main), and the AOT keeps _pf64’s bytes (resources.bin 10,078,970,328 B both).main alone cos ≥
0.9999922, the token gate 16 / 16 on the alphabet (14 tokens + 50 pads) and on the 215-token prompt (22 + 42),
the stop-token check passes._pf64’s graph walked the same way keeps the pads’ columns as its conv state —
the boundary cos falls to 0.8645 while top-1 stays at 95.79 %, above the 93 % bar. Top-1 alone does not catch
a wrong conv state; the boundary cos does.checkpoint() for it (the pin’s InferenceEngine declares it; only its sequential
engine had it), driven the kit’s way (rewind to the shared prefix, then generate), gives turn 2 the full
re-prefill’s 16 greedy tokens on 3 conversations (thinking on, thinking off, and a long one that opens with the
document’s 1,584-token prompt); on the two short ones the first token’s logits are bit-equal. With the conv
state not written back the logits move by up to 9.2 — the check goes red.engine.checkpoint() after a turn’s prompt; its existing rewind to the
longest shared prefix then lands on the checkpoint.main.mlirb carries the export machine’s absolute paths (every op’s source location). The
shipped .aimodel is re-saved after coreai-torch’s strip_debug_info; that is a new asset (a new
main.hash, a new compile-cache entry), so its gates are run again on the stripped bytes.metadata.json points assets.main at the .aimodel (functions main and prefill); on the
Mac the engine specializes it on first load.ios-h19p/ is the static form’s stripped .aimodel compiled for h19p: its metadata.json
points assets.main at the .aimodelc, records the compile command and the checkpoint revision, and drops the
export machine’s checkpoint path (“iPhone” below).</think> into one
answer in 21 runs (0 in 21 on the main-only build; the prompt never reaches prefill): CoreAIKit’s
ChatSession passes it into the reply.extraneous argument label 'capabilities:' in
call (CoreAILanguageModel.swift:51; apple/coreai-models’ main of 2026-10-28 makes that call without the
label). Dropping the label builds it; the engine is unchanged.cpu_only() rejected the tiny model’s stateful dynamic graph on every call with
inferenceFailed(-1), in one run; the 1.7B graph was not tried on the CPU. The CPU reference is the PyTorch
module; the graph’s own reference is its fp32 export compiled for the GPU.torch.export with
Constraints violated (d_32): the externalized SDPA has no lower bound on the KV length and meets the
composite’s _check_is_size(k_len − q_len). Lfm3Model.set_explicit_mask() switches the layers to a bool
mask built from the positions (the composite then runs without its window); a tiny static S=16 graph passes
on the GPU with it. The 1.7B’s S=64 graph: “Chunked prefill” above.lfm3 model type: loading
LiquidAI’s own LFM3-1.7B-MLX-8bit / -4bit stops at ValueError: Model type lfm3 not supported. mlx-vlm
0.7.11 loads them; the card’s comparison with MLX runs through it.The iPhone bundle (ios-h19p/) is the static Neural Engine form (B below), stripped of its source locations and
compiled for the iPhone 18 Pro, and it runs on CoreAIKit’s path. Every run here is on iOS 27.2; the speed columns use
a 768-token prompt and 256 generated tokens (for A, prompt + answer = 1,024 is the cap that coreai-models’ bd8dcf7
puts on the pipelined engine’s dynamic KV on iOS):
| form | phone, engine | prefill / decode tok/s | a 768 + 256 turn | thermal | answers |
|---|---|---|---|---|---|
A. int8hu, main only, compiled ahead of time for the GPU (h18p) |
iPhone 17 Pro, PipelinedBench on a coreai-models pipelined engine with bd8dcf7’s iOS KV cap | 10.0 [9.2–11.0] / 9.25 [8.0–10.7], the 4 warm trials (the cold trial: 12.1 / 10.5) | 104.37 s | nominal → fair in the first run, serious from the second (all 4 warm trials at serious) | the 8 questions’ greedy tokens = the Mac’s on 8 / 8; the token gate 16 / 16 in all 3 runs; stops where the Mac stops |
| B. the static Neural Engine form (h18p) | iPhone 17 Pro, Apple’s StaticShapeEngine (apple/coreai-models’ main) |
339.6 [336.1–340.6] / 32.86 [32.36–33.17], n = 6 | 10.02 s | fair at the start and end of both speed runs (serious in the three correctness runs) | top-1 692 / 721 = 95.98 % teacher-forced vs transformers fp32 (bar 93 %); the 8 questions’ verdicts of transformers bf16, 8 / 8; stops on <\|im_end\|> |
| B, the published bytes (h19p) | iPhone 18 Pro, the StaticShapeEngine CoreAIKit pins |
590.6 [583.7–592.7] / 47.17 [46.28–47.22], n = 6 | 6.71 s | nominal at the start and end of every run | the same tokens as the row above on every prompt (“The kit’s engine against Apple’s”, below) |
main-only build, compiled for h18p with --expect-frequent-reshapes (6.39
GB: the int8 Linears’ fp16 copies), on the pipelined engine with an S=1 prefill. B is
conversion/export_lfm3_ane_static.py over models/ios/lfm3.py: 8-bit palettized weights, the tied head as an
int8 table, contexts up to 4,096; .aimodel 1.76 GB (1.75 GB after the strip), compiled for h18p or h19p 2.15 GB
with each of the 30
transformer functions in a Neural Engine region..aimodel keeps the export’s 138,196 operations in 33
kinds, and its h19p compile has the same 30 Neural Engine programs and the same resources.bin (1,476,692,332 B)
as the compile of the unstripped export, byte for byte; what changes is the location table in main-h19p.mlirb,
the MPSGraph delegate’s two graphs (about 1.2 MB smaller each) and the metadata, and the export machine’s paths go
from 35 hits in 4 files to none. The strip changes nothing the phone computes.0.2.8-zoo, whose StaticShapeEngine differs
from apple/coreai-models’ main by 177 lines (main allocates the step inputs once and resolves input names from a
table; the fork allocates them per call and matches names by substring). For this bundle’s five inputs, two states
and one output, both engines write the same values into the same tensors (checked by reading the code, not by
running it), and on the phones the
tokens agree: the iPhone 18 Pro on the kit’s engine and the iPhone 17 Pro on Apple’s give the same tokens on all
18 prompts (the 17 sequences and the stop-token prompt), teacher-forced and generated, and on the 8 questions.ChatSession(catalog: "lfm3-1.7b") reads the catalog’s ios variant
(engine static-shape), takes ios-h19p/ because Core AI reports the phone’s architecture as h19p
(AIModel.deviceArchitectureName), and loads it on the static-shape engine; each of 6 launches logged that
subtree, the graph main-h19p.mlirb and “Creating static-shape engine”. The bundle was copied into the kit’s
model store at the path a download would use, and the catalog entry came from the kit’s built-in catalog (the live
catalog did not list the model yet). Thinking on, the 8 questions in 3 launches: 8 / 8 each, </think> closed
within the token budget, the reasoning in
.reasoning and out of the answer, the same answers in all 3 (reasoning 47–508 tokens, median 62). Thinking off:
the 8 verdicts of transformers bf16. A tool round trip through KitLanguageModel (engineVariant: .staticShape)
and LanguageModelSession: 3 / 3 with the tool’s value verbatim. A 3-turn chat recalled its first turn; every
turn ended on the stop token; turns 2 and 3 reused the earlier turns’ prompt tokens (32, then 70) and reached their
first token after 0.13 and 0.11 s against 1.14 s on turn 1.<think> with thinking off and reasons for 68 tokens before it answers “Merci” (111 tokens in
all), on the kit’s engine on the 18 Pro and on Apple’s on the 17 Pro alike; transformers bf16 and the Mac bundle
answer it directly. The verdict is still transformers bf16’s.com.apple.developer.kernel.increased-memory-limit entitlement (peak
footprint 2.82 GB in the gate app, up to 2.94 GB through ChatSession); a run without it was not tried.ios-h18p/: on that phone the kit
throws variantNotFound without downloading anything..aimodel specialized on the phone (device JIT) was not tried on the 17 Pro: the device JIT of two other
LFM3 decoders on that phone had ended without an engine, with jetsam events. On an iPhone 18 Pro the int8hu
.aimodel (main only) had not built its engine 300 s after launch (the cause is not isolated); the Mac bundle’s
.aimodel and the int8hu GPU compile for h19p were not run there..aimodel files are named after their weights and did not collide (0 of 3).Mac speed is on the card.