{
  "artifacts": [
    {
      "file": "MNN-Pocket-0.3B_int8_ekv2048.litertlm",
      "sha256": "3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315",
      "size_mb": 305.2
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "taobao-mnn/MNN-Pocket-0.3B rev 087a52fd -> export_hf --use_qkv_norm_rope_composite=True --use_sdpa_composite=True, prefill signatures 128 and 1,024, --cache_length 2048, recipe dynamic_wi8_afp32; the source's tokenizer.json unchanged, the source's chat template with two changes, stop tokens 65512 and 65510, model type generic_model (card L193, L195, L198-L200)",
    "quantization": "int8 weights (one scale per output channel) with float activations, fp16 on the GPU; KV cache of 2,048 tokens; prefill signatures of 128 and 1,024 tokens; the input embedding and the output layer share one int8 table (card L31, L41)",
    "tool": "litert-torch 0.10.0 export_hf (fused QKV and gate/up projections, GPU composites, the attention composites) packed with litert-lm-builder 0.18.1; repro = hf-to-litertlm REPRODUCE.md, entry \"2026-11-07 update\" (card L195, L201)",
    "tool_version": "litert-torch 0.10.0, litert-lm-builder 0.18.1 (litert-torch 0.10.0 calls a builder API that litert-lm-builder 0.19 no longer has); LiteRT-LM 0.19.1 for every measurement (card L194-L195)"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-11-07",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocket-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), one process per run: `Engine(model, cpu, max_num_tokens=2048, enable_benchmark=True); per item: render on a throwaway conversation, then create_conversation(SamplerConfig(top_k=1)).send_message(content, max_output_tokens=1024)`",
            "Device: model=SM-S942Q release=16 build_id=BP4A.251205.006 fingerprint=samsung/m1qjpnx/m1q:16/BP4A.251205.006/S942QOPS1AZH9_SJP1AZH9:user/release-keys",
            "f2-cpu-ga (gate8q, 2026-11-07 00:02 JST): n 8, stopped at the stop token 8, cap hits 0, errors 0, reply text equal to the reference's 8 of 8, answer mismatches 0, divergent 0, first divergence more than 2.0 logits behind 0; run verdict PASS",
            "f2-cpu-gb (gsm20, 2026-11-07 00:03 JST): n 20, stopped at the stop token 18, cap hits 2, errors 0, reply text equal to the reference's 20 of 20, answer mismatches 0, divergent 0, first divergence more than 2.0 logits behind 0, correct 12; run verdict PASS (D15: all 100 decide = False)",
            "XNNPACK, prefill_1024 255 of 436 nodes on its second pass (the rest on the CPU kernels)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "gate8q_n": 8.0,
            "gate8q_stopped": 8.0,
            "gate8q_text_equal": 8.0,
            "gsm20_n": 20.0,
            "gsm20_stopped": 18.0,
            "gsm20_text_equal": 20.0
          },
          "output_match": true,
          "peak_mem_mb": 704.6513,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the device line on the CPU: 8 short questions (each must end before the 1,024-token cap) and the first 20 GSM8K test questions (4-shot chain of thought, greedy, cap 1,024 new tokens), every reply compared with the Mac CPU's reply at 1 thread",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "cpu",
          "context_length": 2048,
          "date": "2026-11-06",
          "decode_tokens_per_s": 87.405,
          "delegated_ops": 2250,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocket-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), the Python API: `Benchmark(model, cpu, prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048); 3 x run on one engine`; protocol: a benchmark script, one process per run, the phone's state and a ready check before each run",
            "Device: model=SM-S942Q release=16 build_id=BP4A.251205.006 fingerprint=samsung/m1qjpnx/m1q:16/BP4A.251205.006/S942QOPS1AZH9_SJP1AZH9:user/release-keys",
            "f2-cpu-t1 (run s26b, 2026-11-06 23:54:59 JST at the run's start, clean; rested 98s since 23:53:21; front com.android.settings (a system app: left alone)): runs 2-3 prefill 778.40 / 616.37, decode 92.17 / 81.04 tok/s, TTFT 1.326 / 1.674 s, Init Total 309.12 ms; VmHWM 1454428 kB (the 1 s sampler), warm runs CPU-capped samples [0, 5]",
            "f2-cpu-t2 (run s26b, 2026-11-06 23:56:43 JST at the run's start, clean; rested 90s since 23:55:13; front com.android.settings (a system app: left alone)): runs 2-3 prefill 776.77 / 615.94, decode 95.25 / 81.16 tok/s, TTFT 1.329 / 1.675 s, Init Total 328.34 ms; VmHWM 1454084 kB (the 1 s sampler), warm runs CPU-capped samples [0, 5]",
            "(the same counts in f2-cpu-t2.err):",
            "VERBOSE: Replacing 255 out of 436 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 284 partitions for subgraph 0 (prefill_128).",
            "VERBOSE: Replacing 255 out of 436 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 284 partitions for subgraph 1 (prefill_1024).",
            "VERBOSE: Replacing 226 out of 406 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 284 partitions for subgraph 2 (decode).",
            "VERBOSE: Replacing 2080 out of 2343 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 114 partitions for subgraph 2 (decode).",
            "VERBOSE: Replacing 2250 out of 2514 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 114 partitions for subgraph 1 (prefill_1024).",
            "plus one XNNPACK line per composite subgraph: odml.cache_update 84, odml.qkv_norm_rope 84, odml.rms_norm 171, odml.sdpa_transposed 84, odml.swiglu 84; the record's counts = the last prefill_1024 line (XNNPACK's second pass over the graph)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_total_ms": 318.73,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_decode_tok_s": 86.604,
            "run1_decode_tok_s_it1": 110.7962,
            "run1_init_total_ms": 309.12,
            "run1_prefill_tok_s": 697.383,
            "run1_ttft_s": 1.5,
            "run1_vmhwm_kb": 1454428.0,
            "run2_decode_tok_s": 88.206,
            "run2_decode_tok_s_it1": 110.3599,
            "run2_init_total_ms": 328.34,
            "run2_prefill_tok_s": 696.352,
            "run2_ttft_s": 1.502,
            "run2_vmhwm_kb": 1454084.0,
            "runs": 2.0
          },
          "output_match": null,
          "peak_mem_mb": 1489.3343,
          "prefill_tokens_per_s": 696.8675,
          "provenance": "measured",
          "runs": true,
          "total_ops": 2514,
          "ttft_ms": 1501.0
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-11-07",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocket-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), one process per run: `Engine(model, gpu, max_num_tokens=2048, enable_benchmark=True); per item: render on a throwaway conversation, then create_conversation(SamplerConfig(top_k=1)).send_message(content, max_output_tokens=1024)`",
            "Device: model=SM-S942Q release=16 build_id=BP4A.251205.006 fingerprint=samsung/m1qjpnx/m1q:16/BP4A.251205.006/S942QOPS1AZH9_SJP1AZH9:user/release-keys",
            "f2-gpu-ga (gate8q, 2026-11-06 23:36 JST): n 8, stopped at the stop token 8, cap hits 0, errors 0, reply text equal to the reference's 8 of 8, answer mismatches 0, divergent 0, first divergence more than 2.0 logits behind 0; run verdict PASS",
            "f2-gpu-gb (gsm20, 2026-11-06 23:37 JST): n 20, stopped at the stop token 17, cap hits 3, errors 0, reply text equal to the reference's 8 of 20, answer mismatches 4, divergent 12, first divergence more than 2.0 logits behind 0, correct 12; run verdict FAIL (D15: all 100 decide = True)",
            "GSM8K-100 on the phone (runs f2-gpu-k1a, f2-gpu-k1b, f2-gpu-k1c, f2-gpu-k1d, f2-gpu-k1e, 2026-11-07 00:21 - 00:30 JST at the runs' starts): 59 / 100 (the source model in bfloat16: 58; paired both / bf16 only / phone only / neither 55 / 3 / 4 / 38, McNemar p 1.0; cap hits 17); rule D15: pass 1 >= 53 PASS, < 53 FAIL; verdict PASS",
            "Every signature whole on LITERT_CL in 1 partition (prefill_128 / prefill_1024 436 of 436 nodes)"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "gate8q_n": 8.0,
            "gate8q_stopped": 8.0,
            "gate8q_text_equal": 8.0,
            "gsm20_n": 20.0,
            "gsm20_stopped": 17.0,
            "gsm20_text_equal": 8.0,
            "gsm8k100_bar": 53.0,
            "gsm8k100_correct": 59.0,
            "gsm8k100_reference_bf16_correct": 58.0
          },
          "output_match": true,
          "peak_mem_mb": 385.8924,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the device line on the GPU: 8 short questions (stop) and the first 20 GSM8K questions compared with the Mac GPU's majority reply; with fewer than 18 of 20 equal and no first divergence more than 2.0 logits behind the reference's first choice, all 100 GSM8K questions on the phone decide (bar 53)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": 2048,
          "date": "2026-11-06",
          "decode_tokens_per_s": 120.0495,
          "delegated_ops": 436,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocket-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), the Python API: `Benchmark(model, gpu, prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048); 3 x run on one engine`; protocol: a benchmark script, one process per run, the phone's state and a ready check before each run",
            "Device: model=SM-S942Q release=16 build_id=BP4A.251205.006 fingerprint=samsung/m1qjpnx/m1q:16/BP4A.251205.006/S942QOPS1AZH9_SJP1AZH9:user/release-keys",
            "f2-gpu-t1 (run s26a, 2026-11-06 23:33:09 JST at the run's start, clean; rested 47s since 23:32:22; front com.android.settings (a system app: left alone)): runs 2-3 prefill 5898.60 / 6050.62, decode 120.01 / 120.12 tok/s, TTFT 0.182 / 0.178 s, Init Total 1539.85 ms; VmHWM 365496 kB (the 1 s sampler), warm runs CPU-capped samples [0, 3]",
            "f2-gpu-t2 (run s26a, 2026-11-06 23:34:05 JST at the run's start, clean; rested 47s since 23:33:18; front com.android.settings (a system app: left alone)): runs 2-3 prefill 5966.97 / 6019.82, decode 120.15 / 119.92 tok/s, TTFT 0.180 / 0.178 s, Init Total 1099.72 ms; VmHWM 384616 kB (the 1 s sampler), warm runs CPU-capped samples [0, 3]",
            "(the same counts in f2-gpu-t2.err):",
            "VERBOSE: Replacing 436 out of 436 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (prefill_128).",
            "VERBOSE: Replacing 436 out of 436 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 1 (prefill_1024).",
            "VERBOSE: Replacing 406 out of 406 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 2 (decode)."
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_total_ms": 1319.785,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_decode_tok_s": 120.062,
            "run1_decode_tok_s_it1": 109.9135,
            "run1_init_total_ms": 1539.85,
            "run1_prefill_tok_s": 5974.614,
            "run1_ttft_s": 0.18,
            "run1_vmhwm_kb": 365496.0,
            "run2_decode_tok_s": 120.037,
            "run2_decode_tok_s_it1": 107.4939,
            "run2_init_total_ms": 1099.72,
            "run2_prefill_tok_s": 5993.394,
            "run2_ttft_s": 0.179,
            "run2_vmhwm_kb": 384616.0,
            "runs": 2.0
          },
          "output_match": null,
          "peak_mem_mb": 393.8468,
          "prefill_tokens_per_s": 5984.004,
          "provenance": "measured",
          "runs": true,
          "total_ops": 436,
          "ttft_ms": 179.5
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__galaxy-s26.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-11-06",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: the PyPI litert-lm 0.19.1 wheel on the Mac (GPU = WebGPU on Metal), one fresh process per pass (greedy: SamplerConfig(top_k=1); a fresh conversation per question; at most 1,024 new tokens); GSM8K test questions 1-100, the prompt = the scorer's 4-shot chain-of-thought prefix + 'Q: <question>\\nA:' as one user turn, scored on the number after the first 'The answer is', else the last number; bar = the source model's score on the same 100 questions in transformers bfloat16 (58) minus 5 = 53",
            "Correct 59 / 58 / 61 of 100 (cap hits 14 / 13 / 12, errors 0); mean 59.33 >= 53; F2's Mac line, completed 2026-11-06 23:04 JST",
            "the measured file was model.litertlm, the file the update uploaded (local and read-back sha256 equal)"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "gsm8k100_bar": 53.0,
            "gsm8k100_mean": 59.3333,
            "gsm8k100_pass1_correct": 59.0,
            "gsm8k100_pass2_correct": 58.0,
            "gsm8k100_pass3_correct": 61.0,
            "gsm8k100_reference_bf16_correct": 58.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "GSM8K-100 (the test split's first 100, 4-shot chain of thought, greedy, cap 1,024 new tokens), three fresh processes; bar 53 = the source model in bfloat16 (58) minus 5",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-06/mnn-pocket-0.3b-int8-ekv2048__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": 2048,
          "date": "2026-11-06",
          "decode_tokens_per_s": 116.025,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: the PyPI litert-lm 0.19.1 wheel (Python 3.13.13; GPU = WebGPU on Metal); `litert-lm benchmark <file> -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu`, a warm-up and 3 runs in each of two processes; the CLI prints the mean of the 3 runs; the cell = the mean of the two processes",
            "window; the measured file was model.litertlm, the file the update uploaded (local and read-back sha256 equal)",
            "os_build: sw_vers read 2026-11-07 at this entry (macOS 27.0, 26A428)",
            "Prefill speed:        1290.67 tokens/s | Decode speed:         121.58 tokens/s | Init time:            2.1102 s | Time to first token:  0.8017 s",
            "Prefill speed:        939.47 tokens/s | Decode speed:         110.47 tokens/s | Init time:            1.3753 s | Time to first token:  2.0509 s"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_s": 1.7428,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_decode_tok_s": 121.58,
            "run1_init_s": 2.1102,
            "run1_prefill_tok_s": 1290.67,
            "run1_ttft_s": 0.8017,
            "run2_decode_tok_s": 110.47,
            "run2_init_s": 1.3753,
            "run2_prefill_tok_s": 939.47,
            "run2_ttft_s": 2.0509,
            "runs": 2.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 1115.07,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 1426.3
        },
        "source": "data/device_runs/0.19.1/2026-11-06/mnn-pocket-0.3b-int8-ekv2048__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-11-06",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: the PyPI litert-lm 0.19.1 wheel on the Mac (GPU = WebGPU on Metal), one fresh process per pass (greedy: SamplerConfig(top_k=1); a fresh conversation per question; at most 1,024 new tokens); GSM8K test questions 1-100, the prompt = the scorer's 4-shot chain-of-thought prefix + 'Q: <question>\\nA:' as one user turn, scored on the number after the first 'The answer is', else the last number; bar = the source model's score on the same 100 questions in transformers bfloat16 (58) minus 5 = 53",
            "Correct 64 / 62 / 63 of 100 (cap hits 13 / 14 / 13, errors 0); mean 63.00 >= 53; F2's Mac GPU chain ended 2026-11-06 22:47:25 JST",
            "the measured file was model.litertlm, the file the update uploaded (local and read-back sha256 equal)"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "gsm8k100_bar": 53.0,
            "gsm8k100_mean": 63.0,
            "gsm8k100_pass1_correct": 64.0,
            "gsm8k100_pass2_correct": 62.0,
            "gsm8k100_pass3_correct": 63.0,
            "gsm8k100_reference_bf16_correct": 58.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "GSM8K-100 (the test split's first 100, 4-shot chain of thought, greedy, cap 1,024 new tokens), three fresh processes; bar 53 = the source model in bfloat16 (58) minus 5",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-06/mnn-pocket-0.3b-int8-ekv2048__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": 2048,
          "date": "2026-11-06",
          "decode_tokens_per_s": 490.94,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: the PyPI litert-lm 0.19.1 wheel (Python 3.13.13; GPU = WebGPU on Metal); `litert-lm benchmark <file> -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend gpu`, a warm-up and 3 runs in each of two processes; the CLI prints the mean of the 3 runs; the cell = the mean of the two processes",
            "window; the measured file was model.litertlm, the file the update uploaded (local and read-back sha256 equal)",
            "os_build: sw_vers read 2026-11-07 at this entry (macOS 27.0, 26A428)",
            "Prefill speed:        18256.92 tokens/s | Decode speed:         490.82 tokens/s | Init time:            1.3859 s | Time to first token:  0.0581 s",
            "Prefill speed:        18280.62 tokens/s | Decode speed:         491.06 tokens/s | Init time:            0.6727 s | Time to first token:  0.0581 s"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_s": 1.0293,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_decode_tok_s": 490.82,
            "run1_init_s": 1.3859,
            "run1_prefill_tok_s": 18256.92,
            "run1_ttft_s": 0.0581,
            "run2_decode_tok_s": 491.06,
            "run2_init_s": 0.6727,
            "run2_prefill_tok_s": 18280.62,
            "run2_ttft_s": 0.0581,
            "runs": 2.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 18268.77,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 58.1
        },
        "source": "data/device_runs/0.19.1/2026-11-06/mnn-pocket-0.3b-int8-ekv2048__mac-studio-m4-max.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-11-07",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocket-devicebench",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Tensor G6, Imagination PowerVR GPU",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), one process per run: `Engine(model, cpu, max_num_tokens=2048, enable_benchmark=True); per item: render on a throwaway conversation, then create_conversation(SamplerConfig(top_k=1)).send_message(content, max_output_tokens=1024)`",
            "Device: model=Pixel 11 Pro release=17 build_id=CD1A.260905.001.B1 fingerprint=google/grizzly/grizzly:17/CD1A.260905.001.B1/16238327:user/release-keys",
            "f2-cpu-ga (gate8q, 2026-11-07 00:35 JST): n 8, stopped at the stop token 8, cap hits 0, errors 0, reply text equal to the reference's 8 of 8, answer mismatches 0, divergent 0, first divergence more than 2.0 logits behind 0; run verdict PASS",
            "f2-cpu-gb (gsm20, 2026-11-07 00:36 JST): n 20, stopped at the stop token 18, cap hits 2, errors 0, reply text equal to the reference's 20 of 20, answer mismatches 0, divergent 0, first divergence more than 2.0 logits behind 0, correct 12; run verdict PASS (D15: all 100 decide = False)",
            "XNNPACK, prefill_1024 255 of 436 nodes on its second pass (the rest on the CPU kernels)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "gate8q_n": 8.0,
            "gate8q_stopped": 8.0,
            "gate8q_text_equal": 8.0,
            "gsm20_n": 20.0,
            "gsm20_stopped": 18.0,
            "gsm20_text_equal": 20.0
          },
          "output_match": true,
          "peak_mem_mb": 709.8982,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the device line on the CPU: 8 short questions (each must end before the 1,024-token cap) and the first 20 GSM8K test questions (4-shot chain of thought, greedy, cap 1,024 new tokens), every reply compared with the Mac CPU's reply at 1 thread",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__pixel-11-pro.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "cpu",
          "context_length": 2048,
          "date": "2026-11-07",
          "decode_tokens_per_s": 76.3625,
          "delegated_ops": 2250,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocket-devicebench",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Tensor G6, Imagination PowerVR GPU",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), the Python API: `Benchmark(model, cpu, prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048); 3 x run on one engine`; protocol: a benchmark script, one process per run, the phone's state and a ready check before each run",
            "Device: model=Pixel 11 Pro release=17 build_id=CD1A.260905.001.B1 fingerprint=google/grizzly/grizzly:17/CD1A.260905.001.B1/16238327:user/release-keys",
            "f2-cpu-t1 (run p11b, 2026-11-07 01:05:06 JST at the run's start, uncapped (thermal 1, no CPU cap, gpu_cooling 0); rested 83s since 01:03:42): runs 2-3 prefill 235.33 / 232.29, decode 79.49 / 69.94 tok/s, TTFT 4.364 / 4.423 s, Init Total 962.38 ms; VmHWM 1300596 kB (the 1 s sampler), warm runs CPU-capped samples [0, 7]",
            "f2-cpu-t2 (run p11b, 2026-11-07 01:06:33 JST at the run's start, uncapped (thermal 1, no CPU cap, gpu_cooling 0); rested 59s since 01:05:32): runs 2-3 prefill 235.68 / 285.14, decode 76.75 / 79.26 tok/s, TTFT 4.358 / 3.604 s, Init Total 964.19 ms; VmHWM 1300808 kB (the 1 s sampler), warm runs CPU-capped samples [0, 7]",
            "(the same counts in f2-cpu-t2.err):",
            "VERBOSE: Replacing 255 out of 436 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 284 partitions for subgraph 0 (prefill_128).",
            "VERBOSE: Replacing 255 out of 436 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 284 partitions for subgraph 1 (prefill_1024).",
            "VERBOSE: Replacing 226 out of 406 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 284 partitions for subgraph 2 (decode).",
            "VERBOSE: Replacing 2080 out of 2343 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 114 partitions for subgraph 2 (decode).",
            "VERBOSE: Replacing 2250 out of 2514 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 114 partitions for subgraph 1 (prefill_1024).",
            "plus one XNNPACK line per composite subgraph: odml.cache_update 84, odml.qkv_norm_rope 84, odml.rms_norm 171, odml.sdpa_transposed 84, odml.swiglu 84; the record's counts = the last prefill_1024 line (XNNPACK's second pass over the graph)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_total_ms": 963.285,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_decode_tok_s": 74.718,
            "run1_decode_tok_s_it1": 77.7454,
            "run1_init_total_ms": 962.38,
            "run1_prefill_tok_s": 233.809,
            "run1_ttft_s": 4.393,
            "run1_vmhwm_kb": 1300596.0,
            "run2_decode_tok_s": 78.007,
            "run2_decode_tok_s_it1": 76.8754,
            "run2_init_total_ms": 964.19,
            "run2_prefill_tok_s": 260.407,
            "run2_ttft_s": 3.981,
            "run2_vmhwm_kb": 1300808.0,
            "runs": 2.0
          },
          "output_match": null,
          "peak_mem_mb": 1332.0274,
          "prefill_tokens_per_s": 247.108,
          "provenance": "measured",
          "runs": true,
          "total_ops": 2514,
          "ttft_ms": 4187.0
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__pixel-11-pro.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-11-07",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocket-devicebench",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Tensor G6, Imagination PowerVR GPU",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), one process per run: `Engine(model, gpu, max_num_tokens=2048, enable_benchmark=True); per item: render on a throwaway conversation, then create_conversation(SamplerConfig(top_k=1)).send_message(content, max_output_tokens=1024)`",
            "Device: model=Pixel 11 Pro release=17 build_id=CD1A.260905.001.B1 fingerprint=google/grizzly/grizzly:17/CD1A.260905.001.B1/16238327:user/release-keys",
            "f2-gpu-ga (gate8q, 2026-11-07 00:32 JST): n 8, stopped at the stop token 8, cap hits 0, errors 0, reply text equal to the reference's 8 of 8, answer mismatches 0, divergent 0, first divergence more than 2.0 logits behind 0; run verdict PASS",
            "f2-gpu-gb (gsm20, 2026-11-07 00:32 JST): n 20, stopped at the stop token 19, cap hits 1, errors 0, reply text equal to the reference's 8 of 20, answer mismatches 4, divergent 12, first divergence more than 2.0 logits behind 0, correct 13; run verdict FAIL (D15: all 100 decide = True)",
            "GSM8K-100 on the phone (runs f2-gpu-k1a, f2-gpu-k1b, f2-gpu-k1c, f2-gpu-k1d, f2-gpu-k1e, 2026-11-07 01:22 - 01:38 JST at the runs' starts): 63 / 100 (the source model in bfloat16: 58; paired both / bf16 only / phone only / neither 55 / 3 / 8 / 34, McNemar p 0.2266; cap hits 15); rule D15: pass 1 >= 53 PASS, < 53 FAIL; verdict PASS",
            "Every signature whole on LITERT_CL in 1 partition (prefill_128 / prefill_1024 436 of 436 nodes)"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "gate8q_n": 8.0,
            "gate8q_stopped": 8.0,
            "gate8q_text_equal": 8.0,
            "gsm20_n": 20.0,
            "gsm20_stopped": 19.0,
            "gsm20_text_equal": 8.0,
            "gsm8k100_bar": 53.0,
            "gsm8k100_correct": 63.0,
            "gsm8k100_reference_bf16_correct": 58.0
          },
          "output_match": true,
          "peak_mem_mb": 504.9713,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the device line on the GPU: 8 short questions (stop) and the first 20 GSM8K questions compared with the Mac GPU's majority reply; with fewer than 18 of 20 equal and no first divergence more than 2.0 logits behind the reference's first choice, all 100 GSM8K questions on the phone decide (bar 53)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__pixel-11-pro.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "gpu",
          "context_length": 2048,
          "date": "2026-11-07",
          "decode_tokens_per_s": 33.283,
          "delegated_ops": 436,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocket-devicebench",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Tensor G6, Imagination PowerVR GPU",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_int8_ekv2048.litertlm` at main 6ea52447: LFS sha256 3d4710d406870fd0a65169503851335256445580999f56a9c219da0e51e49315, 320025488 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: PyPI litert-lm-api 0.19.1 Android wheel on the phone (CPython 3.14.8; liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), the Python API: `Benchmark(model, gpu, prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048); 3 x run on one engine`; protocol: a benchmark script, one process per run, the phone's state and a ready check before each run",
            "Device: model=Pixel 11 Pro release=17 build_id=CD1A.260905.001.B1 fingerprint=google/grizzly/grizzly:17/CD1A.260905.001.B1/16238327:user/release-keys",
            "f2-gpu-t1 (run p11c, 2026-11-07 01:18:05 JST at the run's start, uncapped (thermal 1, no CPU cap, gpu_cooling 0); rested 18s since 01:17:45): runs 2-3 prefill 2864.88 / 2817.28, decode 33.50 / 33.42 tok/s, TTFT 0.387 / 0.393 s, Init Total 1886.58 ms; VmHWM 287840 kB (the 1 s sampler), warm runs CPU-capped samples [0, 8]",
            "f2-gpu-t2 (run p11c, 2026-11-07 01:18:54 JST at the run's start, uncapped (thermal 1, no CPU cap, gpu_cooling 0); rested 18s since 01:18:34): runs 2-3 prefill 2912.41 / 2862.75, decode 33.49 / 32.73 tok/s, TTFT 0.381 / 0.388 s, Init Total 2079.32 ms; VmHWM 281264 kB (the 1 s sampler), warm runs CPU-capped samples [0, 7]",
            "(the same counts in f2-gpu-t2.err):",
            "VERBOSE: Replacing 436 out of 436 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (prefill_128).",
            "VERBOSE: Replacing 436 out of 436 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 1 (prefill_1024).",
            "VERBOSE: Replacing 406 out of 406 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 2 (decode)."
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_total_ms": 1982.95,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_decode_tok_s": 33.458,
            "run1_decode_tok_s_it1": 33.0251,
            "run1_init_total_ms": 1886.58,
            "run1_prefill_tok_s": 2841.079,
            "run1_ttft_s": 0.39,
            "run1_vmhwm_kb": 287840.0,
            "run2_decode_tok_s": 33.108,
            "run2_decode_tok_s_it1": 32.4284,
            "run2_init_total_ms": 2079.32,
            "run2_prefill_tok_s": 2887.577,
            "run2_ttft_s": 0.385,
            "run2_vmhwm_kb": 281264.0,
            "runs": 2.0
          },
          "output_match": null,
          "peak_mem_mb": 294.7482,
          "prefill_tokens_per_s": 2864.328,
          "provenance": "measured",
          "runs": true,
          "total_ops": 436,
          "ttft_ms": 387.5
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-int8-ekv2048__pixel-11-pro.json"
      }
    ]
  },
  "model": {
    "family": "mnn-pocket",
    "id": "mnn-pocket-0.3b-int8-ekv2048",
    "license": "apache-2.0",
    "source_url": "https://huggingface.co/litert-community/MNN-Pocket-0.3B",
    "task": "text-generation"
  },
  "pitfalls": [
    "Context is 2,048 tokens, prompt and reply together (the source model supports 32K in deployment); for a longer conversation take the 4,096-token sibling, which holds the same weights. This file decodes faster than that sibling everywhere but on the Mac GPU, where the two tie (card L37, L186).",
    "Tool calls come back as text: given tool definitions, the model writes the call as a bare JSON object without the `<tool_call>` tags its template describes (the source model in transformers does the same on the card's test prompt), and the file carries no tool-call configuration, so parse the JSON in your app (card L183).",
    "GPU needs LiteRT-LM 0.19.0 or newer, the first release whose GPU kernels run the attention composites: with 0.18.1 the Mac GPU reports a shader error and the reply is a string of symbols, while its CPU runs them (card L185).",
    "Phone GPUs: 16 or 17 of the 20 GSM8K answers equaled the Mac GPU's, under the card's bar of 18; every first difference was within 2 logits of a reference run's first choice, so all 100 GSM8K questions ran on each phone against the bar of 53: 59 (Galaxy S26) and 63 (Pixel 11 Pro) (card L175).",
    "Phone CPUs: both phones lowered their CPU clocks within the first run of every CPU process, so the CPU figures are at lowered clocks; on the Pixel 11 Pro the CPU decodes faster than the GPU and the GPU prefills faster (card L134).",
    "Model type: the file declares generic_model. LiteRT-LM 0.19.1 gives any other type different GPU settings (allow_src_quantized_fc_conv_ops is on for every type except generic_model); the repo's first upload, typed qwen2p5, failed the answer checks on the Galaxy S26 GPU (card L198).",
    "Chat template: LiteRT-LM hands each message's content to the template as a list of text parts and the source template keeps only string content (every user message rendered empty), and the source template errors on an empty message list; the bundled template reads both forms and renders the empty list, giving transformers' text byte for byte on 7 test conversations (card L199).",
    "Sampling: the file carries the source card's example settings (top-k 20, top-p 0.8, temperature 0.8); every check on the card is greedy (card L41, L160).",
    "No int4 file: none of the int4 builds passed the Mac checks on both the CPU and the GPU, including one with the authors' quantization-aware int4 values (card L184)."
  ],
  "schema_version": "1.2"
}
