{ "artifacts": [ { "file": "MiniCPM5-2B_int8.litertlm", "sha256": "58da116a6eb563185eb5e8d0b1f1e1116b632a516cadc0285b6da35b9f886a86", "size_mb": 2482.68 } ], "benchmarks": [], "conversion": { "command": "convert_minicpm5_2b.sh (USE_JINJA=1 with the verbatim chat_template.jinja, CACHE=4096, PREFILL=1024,256,64,16,4,1, EXTERNALIZE_EMBEDDER=1, HF_HUB_OFFLINE=1, recipe BOCTAV4 | dynamic_wi8_afp32) -> int4: minicpm52b_work/fix_zero_scales_inplace.py (in-place scale-word patch, keeps the embedder section) ; int8: scripts/set_activation_type.py --type fp32 (litert-lm 0.17.0 unpack/pack, only model.toml gains prefer_activation_type) (FINDINGS Rail; card Conversion notes)", "quantization": "dynamic int8 on linears + embedding (dynamic_wi8_afp32, externalized embedder section); prefer_activation_type = fp32 declared in the bundle for the GPU executor (metadata-only repack of the fp16-default export: every section byte-identical, only model.toml differs); 0 zero scales (per-channel quantization of the 13 dead rows is fine)", "tool": "litert-torch export on the jinja path with the checkpoint's chat_template.jinja byte-identical (released stack only; repro = hf-to-litertlm `bash scripts/reproduce_llm.sh minicpm5-2b`; lane script minicpm52b_work/convert_minicpm5_2b.sh -> scripts/export_simple_template.py)", "tool_version": "litert-torch 0.9.3 / litert-converter 0.4.0 / ai-edge-quantizer 0.9.0 / litert-lm-builder 0.16.1 / transformers 5.14.1 (HF card Conversion notes; FINDINGS toolchain ~/venvs/lt093ctl, the granite-4.2-3b rail); card-grade bench and toggle probes on litert-lm 0.17.0" }, "cross_runtime": [], "delegation": null, "device": { "records": [ { "device": "galaxy-s26", "run": { "accelerator": "cpu", "context_length": null, "date": "2026-09-08", "decode_tokens_per_s": 11.67, "delegated_ops": 2280, "env": { "device": "Galaxy S26 (SM-S942Q)", "machine_label": "galaxy-s26-cl-pinned", "os_build": "Android 16", "runtime": "litert-lm", "runtime_version": "0.16.0", "soc": "Qualcomm SM8850", "vendor_sdk": null }, "error": null, "evidence": [ "VERBOSE: Replacing 1699 out of 1873 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 249 partitions for subgraph 0 (prefill_1024).", "VERBOSE: Replacing 1699 out of 1873 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 249 partitions for subgraph 1 (prefill_256).", "VERBOSE: Replacing 1699 out of 1873 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 249 partitions for subgraph 2 (prefill_64).", "VERBOSE: Replacing 1699 out of 1873 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 249 partitions for subgraph 3 (prefill_16).", "results block: prefill=102.76 decode=11.67 tokens/s" ], "failure_class": null, "full_delegation": false, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 2175.0, "init_s": 0.26913, "prefill_tokens": 203.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 102.76, "provenance": "measured", "runs": true, "total_ops": 2371, "ttft_ms": 2060.0 }, "source": "data/device_runs/0.16.0/2026-09-08/minicpm5-2b-int8__galaxy-s26.json" }, { "device": "galaxy-s26", "run": { "accelerator": "gpu", "context_length": null, "date": "2026-09-08", "decode_tokens_per_s": 10.86, "delegated_ops": 1873, "env": { "device": "Galaxy S26 (SM-S942Q)", "machine_label": "galaxy-s26-cl-pinned", "os_build": "Android 16", "runtime": "litert-lm", "runtime_version": "0.16.0", "soc": "Qualcomm SM8850", "vendor_sdk": null }, "error": null, "evidence": [ "VERBOSE: Replacing 1873 out of 1873 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (prefill_1024).", "VERBOSE: Replacing 1873 out of 1873 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 1 (prefill_256).", "VERBOSE: Replacing 1873 out of 1873 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 2 (prefill_64).", "VERBOSE: Replacing 1873 out of 1873 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 3 (prefill_16).", "results block: prefill=150.24 decode=10.86 tokens/s" ], "failure_class": null, "full_delegation": true, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 2699.0, "init_s": 6.34923, "prefill_tokens": 203.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 150.24, "provenance": "measured", "runs": true, "total_ops": 1873, "ttft_ms": 1440.0 }, "source": "data/device_runs/0.16.0/2026-09-08/minicpm5-2b-int8__galaxy-s26.json" }, { "device": "mac-studio-m4-max", "run": { "accelerator": "cpu", "context_length": null, "date": "2026-09-08", "decode_tokens_per_s": 30.02, "delegated_ops": null, "env": { "device": "Mac Studio (M4 Max)", "machine_label": "mac-studio-m4-max", "os_build": null, "runtime": "litert-lm", "runtime_version": "0.17.0", "soc": null, "vendor_sdk": null }, "error": null, "evidence": [ "results block: prefill=160.87 decode=30.02 tokens/s" ], "failure_class": null, "full_delegation": null, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 256.0, "prefill_tokens": 256.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 160.87, "provenance": "measured", "runs": true, "total_ops": null, "ttft_ms": 1624.7 }, "source": "data/device_runs/0.17.0/2026-09-08/minicpm5-2b-int8__mac-studio-m4-max.json" }, { "device": "mac-studio-m4-max", "run": { "accelerator": "gpu", "context_length": null, "date": "2026-09-08", "decode_tokens_per_s": 74.65, "delegated_ops": null, "env": { "device": "Mac Studio (M4 Max)", "machine_label": "m4max-quiet-300s-gpu-rest", "os_build": null, "runtime": "litert-lm", "runtime_version": "0.17.0", "soc": null, "vendor_sdk": null }, "error": null, "evidence": [ "results block: prefill=1405.16 decode=74.65 tokens/s" ], "failure_class": null, "full_delegation": null, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 256.0, "prefill_tokens": 256.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 1405.16, "provenance": "measured", "runs": true, "total_ops": null, "ttft_ms": 195.6 }, "source": "data/device_runs/0.17.0/2026-09-08/minicpm5-2b-int8__mac-studio-m4-max.json" } ] }, "model": { "family": "minicpm5", "id": "minicpm5-2b-int8", "license": "apache-2.0", "source_url": "https://huggingface.co/mlboydaisuke/MiniCPM5-2B-LiteRT", "task": "text-generation" }, "pitfalls": [ "int8 declares fp32 activations in-bundle: with the runtime's default fp16 GPU activations the int8 model's reasoning on the rhyme gate question ran 2021 tokens without closing (7/8 degenerate, fails the gate); with fp32 declared it closes in ~450 tokens and passes 8/8, at ~14% GPU decode (and ~18% prefill) cost on identical weights. On a ten-question thinking-on subset the two dtypes were mixed (fp16 finished 10/10, fp32act 7/10 with two cap hits) — a measured trade for the ship gate, not a cure (HF card Conversion notes; FINDINGS int8 gates + attribution item 3).", "int8 is the file for reasoning that has to complete: its thinking chains are ~3-4x shorter than int4's on the same questions and terminate where int4 runs into the budget; GSM8K thinking-off 91 vs bf16 92 (HF card).", "int8's main weight section is 2,330,645,824 bytes — above the single-section mmap ceiling of default-entitlement iOS apps — so it is a desktop / Android build (no iPhone row; the Spark-X2.5-4B int8 lane later showed a 4.24 GB section loading under the Increased Memory Limit entitlement, so the ceiling is entitlement-dependent) (HF card; FINDINGS int8 build).", "Galaxy S26 int8 fp32act: GPU gate PASS with full delegation, GPU bench prefill 150-160 / decode 10.9-12.8 tok/s at 0.94-1.10 GB peak VmHWM; CPU 103-157 / 11.7 tok/s at 2.90 GB — the card must say 'fp32 activations declared in-bundle' next to these rows (FINDINGS S26 int8 fp32act).", "Requires litert-lm >= 0.16 (thought channel + ThinkingConfig); measured on 0.17.0. The bundle carries the model's own chat_template.jinja verbatim (9060 chars, byte-compared) on the runtime's jinja path — the litert-community/MiniCPM5-1B packaging — so enable_thinking and the tool-calling format stay reachable; the thought channel (/) is auto-declared by the jinja path and was read out of the built bundle, never assumed (HF card; FINDINGS Rail + int4 header).", "Start token is correct for this family: the template's own {{ bos_token }} renders empty at runtime and the engine prepends the metadata start token, so the model sees exactly one as upstream (verified on the official 1B artifact 2026-08-01); stops are (1) and <|im_end|> (130073). The 14 'X<|im_end|>\\n' string stops in the metadata are emitted by litert-torch 0.9.3's own export_hf builder (the tokenizer has no such pieces) — a converter artifact, harmless (FINDINGS premises + int4 header).", "Thinking-OFF baseline: the bf16 PyTorch model scores 8/8 on the 8-question gate with thinking on or unset but 6/8 with thinking forced off ('opposite of hot' -> 'Cool.', rhyme -> 'Green.') — read any thinking-off result against 6/8, not 8/8; both toggle paths (ThinkingConfig and the 0.14-era extra_context) reach the verbatim template on 0.17.0 (HF card Correctness; FINDINGS bf16 oracle + toggle probe).", "GSM8K thinking OFF (the official 1B card protocol; first 100 test questions, greedy, 0-shot CoT prompt, max 2048): bf16 92 / int8 CPU 91 / int4 CPU 86 / int4 GPU fp16 87 / int4 GPU fp32act 88 — int8 at parity (7 of its 9 losses are bf16's own), int4 ~5 points down, and the GPU's fp16 activations cost nothing measurable in no-think mode (HF card; FINDINGS).", "In thinking mode int4's chains are ~4.5x longer than bf16's (median ~13.7k vs ~3k chars on a 10-question subset) and close 0/10 within 3584 tokens on the fp32 CPU reference, while int8 on CPU reproduces bf16 question for question (9/10 finished, same cap hit) — the thinking-length inflation is int4 quantization damage; int4 for direct answers / short thinking, int8 when the reasoning has to complete (HF card; FINDINGS Thinking-ON attribution).", "Galaxy S26 (litert_lm_advanced_main v0.16.0 kit): both files generate correctly on GPU and CPU with full OpenCL delegation (1873/1873 nodes on every prefill signature and decode, zero rejected ops) and the runtime separates the reasoning on-device ([thought] … [/thought], then the answer); on Adreno the int4 GPU decode edge over the same-device CPU is ~1.1x (prefill 5-10x), and the int8 file's fp32 activations bring its GPU decode level with its CPU (11-13 vs 11.7 tok/s) at ~2.6x lower prefill than the int4 file's (HF card Performance; FINDINGS S26 sections).", "Card-grade Mac bench (litert-lm 0.17.0 CLI, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, CPU cells first, >= 300 s rest before every GPU cell, gate_backend.py confirmed real generation on every cell); the earlier gate-time decode figures were taken under load and are not speed claims (FINDINGS Card-grade Mac bench + int8 gates note).", "Block-128 int4 was built and rejected: 7/8 on the CPU gate (the 'merci' question never closes ), GSM8K thinking-off 79 vs 87, and the thinking-length problem does not move (12.9k vs 13.7k chars); the 'reasoning ships prefer block128' heuristic does not apply here, as on granite-4.2 (HF card Conversion notes; FINDINGS block-128 A/B).", "Embedder externalised (130560x2048 table in its own section); tokenizer embedded as the upstream tokenizer.json (byte-level BPE, 130,072 base + 510 added tokens; =8 / =9 plain added tokens, /think and /no_think special) (HF card; FINDINGS premises).", "Local bundles, raw exports, compile caches and the HF source cache were deleted at close-out on 2026-09-08 after re-reading the remote sizes; regeneration = the reproduce script (int4 comes out zero-scale-fixed; the int8 ship additionally needs the fp32 activation repack) or a Hub download; two ~4 GB xnnpack caches remain on the S26 (FINDINGS Close-out)." ], "schema_version": "1.2" }