{
  "artifacts": [
    {
      "file": "mnn-pocket-0.3b-dec_L768_fp16w.tflite",
      "sha256": "7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034",
      "size_mb": 466.233
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "the repository's conversion/ (conversion/README.md) holds the scripts and REPRODUCE.md every check (README L150); the card names the tools, not a command line",
    "quantization": "float16 linear-layer weights, the LoRA adapter folded into the weights first (README L20)",
    "tool": "litert-torch",
    "tool_version": "0.10.0 (for LiteRT 2.3.0)"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "cpu_xnnpack",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2376,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocketdec-capi-runner",
            "os_build": "Android 16 (BP4A.251205.006.S942QOPS1AZH9)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "(tag s26_xt4_L768, run r7): run DONE rc 0, start 2026-11-08 06:21:58, 124 questions",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 12/12 OK",
            "Device: SM-S942Q 16 BP4A.251205.006.S942QOPS1AZH9",
            "Extras: rest=30 args=--cpu-threads,4; the build 4946d828 passes --cpu-threads to LiteRT (the r5 probes: threads_in_calls_max 4 at --cpu-threads 4)",
            "runner.replacing: VERBOSE: Replacing 2376 out of 2432 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 57 partitions for subgraph 0 (main).",
            "line PASS: 124 questions, same winner 119/119 (not near ties), near ties kept 5/5, max |Δp| 0.000908779, mean |Δp| 9.88097e-05, control n/a (the file holds no request of the control pair), non-finite read-outs 0, winner = the Mac CPU's on the same file 119/119",
            "Run memory (11 samples): VmHWM_max_kB 1692488 VmSwap_max_kB 0 MemAvailable_min_kB 5981712 kgsl_page_alloc_max_B 197558272"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "compile_ms": 541.6,
            "line_bar_control_abs_dp": 0.02,
            "line_bar_max_abs_dp": 0.02,
            "line_bar_mean_abs_dp": 0.002,
            "line_bar_near_tie_gap": 0.02,
            "line_mac_same_file_non_near_tie": 119.0,
            "line_max_abs_dp": 0.0009087786688101729,
            "line_mean_abs_dp": 9.880967733262983e-05,
            "line_near_tie": 5.0,
            "line_non_near_tie": 119.0,
            "line_nonfinite_questions": 0.0,
            "line_pass": 1.0,
            "line_questions": 124.0,
            "line_winner_equal_mac_non_near_tie": 119.0,
            "line_winner_equal_near_tie": 5.0,
            "line_winner_equal_non_near_tie": 119.0,
            "threads": 4.0,
            "vmhwm_max_kb": 1692488.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the line: 124 questions of the L768 set against the source model's reference; the C-API runner build 4946d828 (LiteRT 2.3.0), CPU XNNPACK, 4 threads (--cpu-threads 4)",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "cpu_xnnpack",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2376,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocketdec-capi-runner",
            "os_build": "Android 16 (BP4A.251205.006.S942QOPS1AZH9)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "673.48 ms (min 665.33, max 684.20; median of 8 after 2 warm-ups, write+run+readback; run only 672.59 ms; load 0.53 s; Vm… (the cell's value; parity cell: PASS; gate PASS, timing run = check run (5/5 q, dumps equal))",
            "(tag s26_xt4t_d_L768_rt1, run r7): start 2026-11-08 06:54:51, request F2/customer_service_000006 (586 tokens), write_run_read_ms median 673.48 min 665.33 max 684.201 n 8, run_only_ms median 672.5875",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 12/12 OK; extras: rest=180 sample=1 args=--cpu-threads,4",
            "Device: SM-S942Q 16 BP4A.251205.006.S942QOPS1AZH9",
            "clock samples: 8 samples, 6 capped (policy6 3398400/4742400, policy6 3648000/4742400, policy6 3763200/4742400, policy6 3859200/4742400), GPU level max 0",
            "parity: [\"s26_xt4t_d_L768_rt1\"]: gate PASS, timing run = check run (5/5 questions equal in every digit, max |Δp| between 0.0), against the check run",
            "VERBOSE: Replacing 2376 out of 2432 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 57 partitions for subgraph 0 (main).",
            "the cell's takes (take id: median, capped): 600: 712.1 ms capped; 601: 673.5 ms capped; the cell is take 601, the fastest take (every take capped)",
            "threads: the build 4946d828 passes --cpu-threads to LiteRT"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": 673.48,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "capped": 1.0,
            "capped_samples": 6.0,
            "clock_samples": 8.0,
            "compile_ms": 528.7,
            "latency_max_ms": 684.201,
            "latency_min_ms": 665.33,
            "request_questions": 5.0,
            "request_tokens": 586.0,
            "run_only_p50_ms": 672.5875,
            "threads": 4.0,
            "timed_calls": 8.0,
            "vmhwm_max_kb": 1691616.0,
            "warmup_calls": 2.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "timing: the typed-decisions test row `customer_service_000006` (586 tokens, 5 questions), the median of 8 calls after 2 warm-up calls in one process, a call = write the four inputs, run, read `hidden` back; the C-API runner build 4946d828 (LiteRT 2.3.0), CPU XNNPACK, 4 threads (--cpu-threads 4)",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu_mldrift",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2432,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocketdec-capi-runner",
            "os_build": "Android 16 (BP4A.251205.006.S942QOPS1AZH9)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "(tag s26_ga_L768, run r7): run DONE rc 0, start 2026-11-08 06:22:56, 124 questions",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 12/12 OK",
            "Device: SM-S942Q 16 BP4A.251205.006.S942QOPS1AZH9",
            "runner.replacing: VERBOSE: Replacing 2432 out of 2432 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).",
            "line PASS: 124 questions, same winner 119/119 (not near ties), near ties kept 3/5, max |Δp| 0.00482959, mean |Δp| 0.000794907, control n/a (the file holds no request of the control pair), non-finite read-outs 0, winner = the Mac CPU's on the same file 119/119",
            "Run memory (4 samples): VmHWM_max_kB 2066236 VmSwap_max_kB 0 MemAvailable_min_kB 6633168 kgsl_page_alloc_max_B 783020032"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "compile_ms": 1414.5,
            "line_bar_control_abs_dp": 0.02,
            "line_bar_max_abs_dp": 0.02,
            "line_bar_mean_abs_dp": 0.002,
            "line_bar_near_tie_gap": 0.02,
            "line_mac_same_file_non_near_tie": 119.0,
            "line_max_abs_dp": 0.004829591027856184,
            "line_mean_abs_dp": 0.0007949073160554349,
            "line_near_tie": 5.0,
            "line_non_near_tie": 119.0,
            "line_nonfinite_questions": 0.0,
            "line_pass": 1.0,
            "line_questions": 124.0,
            "line_winner_equal_mac_non_near_tie": 119.0,
            "line_winner_equal_near_tie": 3.0,
            "line_winner_equal_non_near_tie": 119.0,
            "vmhwm_max_kb": 2066236.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the line: 124 questions of the L768 set against the source model's reference; the C-API runner build 4946d828 (LiteRT 2.3.0), GPU OpenCL, GpuOptions precision FP16_WITH_FP32_ACCUM",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu_mldrift",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2432,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocketdec-capi-runner",
            "os_build": "Android 16 (BP4A.251205.006.S942QOPS1AZH9)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "(tag s26_gf_L768, run r7): run DONE rc 0, start 2026-11-08 06:22:31, 124 questions",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 12/12 OK",
            "Device: SM-S942Q 16 BP4A.251205.006.S942QOPS1AZH9",
            "runner.replacing: VERBOSE: Replacing 2432 out of 2432 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).",
            "line PASS: 124 questions, same winner 119/119 (not near ties), near ties kept 5/5, max |Δp| 0.000909005, mean |Δp| 9.89258e-05, control n/a (the file holds no request of the control pair), non-finite read-outs 0, winner = the Mac CPU's on the same file 119/119",
            "Run memory (7 samples): VmHWM_max_kB 2543652 VmSwap_max_kB 0 MemAvailable_min_kB 5261024 kgsl_page_alloc_max_B 1308381184"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "compile_ms": 3160.3,
            "line_bar_control_abs_dp": 0.02,
            "line_bar_max_abs_dp": 0.02,
            "line_bar_mean_abs_dp": 0.002,
            "line_bar_near_tie_gap": 0.02,
            "line_mac_same_file_non_near_tie": 119.0,
            "line_max_abs_dp": 0.0009090049141375345,
            "line_mean_abs_dp": 9.892578785332496e-05,
            "line_near_tie": 5.0,
            "line_non_near_tie": 119.0,
            "line_nonfinite_questions": 0.0,
            "line_pass": 1.0,
            "line_questions": 124.0,
            "line_winner_equal_mac_non_near_tie": 119.0,
            "line_winner_equal_near_tie": 5.0,
            "line_winner_equal_non_near_tie": 119.0,
            "vmhwm_max_kb": 2543652.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the line: 124 questions of the L768 set against the source model's reference; the C-API runner build 4946d828 (LiteRT 2.3.0), GPU OpenCL, GpuOptions precision FP32",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu_mldrift",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2432,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocketdec-capi-runner",
            "os_build": "Android 16 (BP4A.251205.006.S942QOPS1AZH9)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "247.84 ms (min 245.92, max 252.22; median of 8 after 2 warm-ups, write+run+readback; run only 1.15 ms; load 1.35 s; VmHW… (the cell's value; parity cell: PASS; gate PASS, timing run = check run (5/5 q, dumps equal))",
            "(tag s26_gat_d_L768_rt1, run r7): start 2026-11-08 06:51:45, request F2/customer_service_000006 (586 tokens), write_run_read_ms median 247.837 min 245.918 max 252.218 n 8, run_only_ms median 1.149",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 12/12 OK; extras: rest=180 sample=1",
            "Device: SM-S942Q 16 BP4A.251205.006.S942QOPS1AZH9",
            "clock samples: 4 samples, 0 capped, GPU level max 0",
            "parity: [\"s26_gat_d_L768_rt1\"]: gate PASS, timing run = check run (5/5 questions equal in every digit, max |Δp| between 0.0), against the check run",
            "VERBOSE: Replacing 2432 out of 2432 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).",
            "the cell's takes (take id: median, capped): 595: 255.7 ms capped; 596: 247.8 ms; the cell is take 596, the fastest uncapped take",
            "other rows of this file, request and backend, not takes of this cell: 597: GPU precision fp32; 598: GPU precision fp32"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": 247.837,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "capped": 0.0,
            "capped_samples": 0.0,
            "clock_samples": 4.0,
            "compile_ms": 1353.4,
            "latency_max_ms": 252.218,
            "latency_min_ms": 245.918,
            "request_questions": 5.0,
            "request_tokens": 586.0,
            "run_only_p50_ms": 1.149,
            "timed_calls": 8.0,
            "vmhwm_max_kb": 2065356.0,
            "warmup_calls": 2.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "timing: the typed-decisions test row `customer_service_000006` (586 tokens, 5 questions), the median of 8 calls after 2 warm-up calls in one process, a call = write the four inputs, run, read `hidden` back; the C-API runner build 4946d828 (LiteRT 2.3.0), GPU OpenCL, GpuOptions precision FP16_WITH_FP32_ACCUM",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu_mldrift",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2432,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocketdec-capi-runner",
            "os_build": "Android 16 (BP4A.251205.006.S942QOPS1AZH9)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "352.43 ms (min 350.58, max 357.98; median of 8 after 2 warm-ups, write+run+readback; run only 1.24 ms; load 1.62 s; VmHW… (the cell's value; parity cell: PASS; gate PASS, timing run = check run (5/5 q, dumps equal))",
            "(tag s26_gft_d_L768_rt1, run r7): start 2026-11-08 06:48:33, request F2/customer_service_000006 (586 tokens), write_run_read_ms median 352.4285 min 350.582 max 357.975 n 8, run_only_ms median 1.242",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 12/12 OK; extras: rest=180 sample=1",
            "Device: SM-S942Q 16 BP4A.251205.006.S942QOPS1AZH9",
            "clock samples: 5 samples, 0 capped, GPU level max 0",
            "parity: [\"s26_gft_d_L768_rt1\"]: gate PASS, timing run = check run (5/5 questions equal in every digit, max |Δp| between 0.0), against the check run",
            "VERBOSE: Replacing 2432 out of 2432 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).",
            "the cell's takes (take id: median, capped): 597: 355 ms capped; 598: 352.4 ms; the cell is take 598, the fastest uncapped take",
            "other rows of this file, request and backend, not takes of this cell: 595: GPU precision fp16acc32; 596: GPU precision fp16acc32"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": 352.4285,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "capped": 0.0,
            "capped_samples": 0.0,
            "clock_samples": 5.0,
            "compile_ms": 1618.9,
            "latency_max_ms": 357.975,
            "latency_min_ms": 350.582,
            "request_questions": 5.0,
            "request_tokens": 586.0,
            "run_only_p50_ms": 1.242,
            "timed_calls": 8.0,
            "vmhwm_max_kb": 2543396.0,
            "warmup_calls": 2.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "timing: the typed-decisions test row `customer_service_000006` (586 tokens, 5 questions), the median of 8 calls after 2 warm-up calls in one process, a call = write the four inputs, run, read `hidden` back; the C-API runner build 4946d828 (LiteRT 2.3.0), GPU OpenCL, GpuOptions precision FP32",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__galaxy-s26.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu_xnnpack",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Name mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16, arm cpu4, child rc 0",
            "Runtime: api CompiledModel.from_file, accelerator CPU (XNNPACK), threads 4, is_fully_accelerated False (the API states no node count), compile 0.13 s",
            "date: the record's first commit, author date 2026-11-08 06:16:36 +0900, after the run's START 05:47:24 (the run log has times only and is not in git)",
            "line PASS: 2362 questions, same winner 2276/2276 (not near ties), near ties kept 86/86, max |Δp| 0.00209024, mean |Δp| 0.000110692, control 0.0401, non-finite read-outs 0"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 20.0,
            "arm_files_run": 20.0,
            "compile_s": 0.13,
            "line_bar_control_abs_dp": 0.02,
            "line_bar_max_abs_dp": 0.02,
            "line_bar_mean_abs_dp": 0.002,
            "line_bar_near_tie_gap": 0.02,
            "line_control_abs_dp": 0.04007461881503663,
            "line_max_abs_dp": 0.002090235550811559,
            "line_mean_abs_dp": 0.00011069246010478091,
            "line_near_tie": 86.0,
            "line_non_near_tie": 2276.0,
            "line_nonfinite_questions": 0.0,
            "line_pass": 1.0,
            "line_questions": 2362.0,
            "line_winner_equal_near_tie": 86.0,
            "line_winner_equal_non_near_tie": 2276.0,
            "threads": 4.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the line: 2362 questions (the requests that fit L768) against the source model's reference; Python CompiledModel (ai-edge-litert 2.3.0), CPU XNNPACK, 4 threads",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu_xnnpack",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "221.45 ms (min 220.41, max 229.34, p90 222.93; median of 20 after 5 warm-ups, write+run+readback; compile 0.12 s) (the cell's value; parity cell: PASS (gate PASS: 2276/2276 non-near-tie, max|Δp| 0.0021, mean 0.00011, 2362 q; e)",
            "(tag cpu4_d_L768_B2, started 2026-11-08 06:07:09 JST, rule \"line L6: 5 warm-ups + median of 20, host write + run + read back; host ms apart\", ai-edge-litert 2.3.0): row file mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16, arm cpu4, request d, ms_write_run_read median 221.45497899327893 min 220.40854199440219 max 229.3410410056822 p90 222.92733332142234 n 20, winners_equal_R True",
            "The record is listed, its graph -> (488880528 B, sha256 7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034) = the published mnn-pocket-0.3b-dec_L768_fp16w.tflite (equal_published true)",
            "the line on this file and arm, line PASS: 2362 questions, same winner 2276/2276 (not near ties), near ties kept 86/86, max |Δp| 0.00209024, mean |Δp| 0.000110692, control 0.0401, non-finite read-outs 0",
            "the cell's takes (take id: median): 607: 222.2 ms; 608: 221.5 ms; the cell is take 608, the fastest",
            "the same request on the L1024 and L768 files one run after the other, L1024, L768, L768, L1024 (abba_* metrics)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": 221.45497899327893,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "abba_a1_l1024_ms": 392.628687506658,
            "abba_a2_l1024_ms": 382.03543750569224,
            "abba_b1_l768_ms": 222.24154201103374,
            "abba_b2_l768_ms": 221.45497899327893,
            "arm_files_passed": 20.0,
            "arm_files_run": 20.0,
            "compile_s": 0.12,
            "latency_max_ms": 229.3410410056822,
            "latency_min_ms": 220.40854199440219,
            "latency_p90_ms": 222.92733332142234,
            "request_questions": 5.0,
            "request_tokens": 586.0,
            "threads": 4.0,
            "timed_calls": 20.0,
            "warmup_calls": 5.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "timing: the typed-decisions test row `customer_service_000006` (586 tokens, 5 questions), the median of 20 calls after 5 warm-up calls in one process, a call = write the four inputs, run, read `hidden` back; Python CompiledModel (ai-edge-litert 2.3.0), CPU XNNPACK, 4 threads",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu_metal_mac",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Name mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16, arm gpu_f32, child rc 0",
            "Runtime: api CompiledModel.from_file, accelerator GPU (Metal), gpu_options {\"enforce_f32\": true}, is_fully_accelerated True (the API states no node count), compile 2.37 s",
            "date: the record's first commit, author date 2026-11-08 06:16:36 +0900, after the run's START 05:51:03 (the run log has times only and is not in git)",
            "line PASS: 2362 questions, same winner 2276/2276 (not near ties), near ties kept 86/86, max |Δp| 0.00208065, mean |Δp| 0.000110706, control 0.0401, non-finite read-outs 0"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 20.0,
            "arm_files_run": 20.0,
            "compile_s": 2.37,
            "line_bar_control_abs_dp": 0.02,
            "line_bar_max_abs_dp": 0.02,
            "line_bar_mean_abs_dp": 0.002,
            "line_bar_near_tie_gap": 0.02,
            "line_control_abs_dp": 0.040074430422127605,
            "line_max_abs_dp": 0.002080653409673283,
            "line_mean_abs_dp": 0.00011070613392594623,
            "line_near_tie": 86.0,
            "line_non_near_tie": 2276.0,
            "line_nonfinite_questions": 0.0,
            "line_pass": 1.0,
            "line_questions": 2362.0,
            "line_winner_equal_near_tie": 86.0,
            "line_winner_equal_non_near_tie": 2276.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the line: 2362 questions (the requests that fit L768) against the source model's reference; Python CompiledModel (ai-edge-litert 2.3.0), GPU Metal, GpuOptions(enforce_f32=True)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu_metal_mac",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "59.80 ms (min 59.45, max 60.04, p90 59.96; median of 20 after 5 warm-ups, write+run+readback; compile 2.12 s) (the cell's value; parity cell: PASS (gate PASS: 2276/2276 non-near-tie, max|Δp| 0.0021, mean 0.00011, 2362 q; e)",
            "(tag gpu_f32_d_L768_B1, started 2026-11-08 06:07:29 JST, rule \"line L6: 5 warm-ups + median of 20, host write + run + read back; host ms apart\", ai-edge-litert 2.3.0): row file mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16, arm gpu_f32, request d, ms_write_run_read median 59.80418750550598 min 59.45179201080464 max 60.038666997570544 p90 59.955753781832755 n 20, winners_equal_R True",
            "The record is listed, its graph -> (488880528 B, sha256 7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034) = the published mnn-pocket-0.3b-dec_L768_fp16w.tflite (equal_published true)",
            "the line on this file and arm, line PASS: 2362 questions, same winner 2276/2276 (not near ties), near ties kept 86/86, max |Δp| 0.00208065, mean |Δp| 0.000110706, control 0.0401, non-finite read-outs 0",
            "the cell's takes (take id: median): 613: 59.8 ms; 614: 60.34 ms; 615: 60.35 ms; 616: 60.05 ms; the cell is take 613, the fastest",
            "REPRODUCE.md's L1024 / L768 order table on this arm is the session; this cell's take is another session's run of the L768 file (the fastest take)"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": 59.80418750550598,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 20.0,
            "arm_files_run": 20.0,
            "compile_s": 2.12,
            "latency_max_ms": 60.038666997570544,
            "latency_min_ms": 59.45179201080464,
            "latency_p90_ms": 59.955753781832755,
            "request_questions": 5.0,
            "request_tokens": 586.0,
            "timed_calls": 20.0,
            "warmup_calls": 5.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "timing: the typed-decisions test row `customer_service_000006` (586 tokens, 5 questions), the median of 20 calls after 5 warm-up calls in one process, a call = write the four inputs, run, read `hidden` back; Python CompiledModel (ai-edge-litert 2.3.0), GPU Metal, GpuOptions(enforce_f32=True)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__mac-studio-m4-max.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "cpu_xnnpack",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2376,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocketdec-capi-runner",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Google Tensor G6",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "(tag p11p_xt5_L768, run r7): run DONE rc 0, start 2026-11-08 07:04:45, 124 questions",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 2/2 OK",
            "Device: Pixel 11 Pro 17 CD1A.260905.001.B1",
            "Extras: rest=30 args=--cpu-threads,5; the build 4946d828 passes --cpu-threads to LiteRT (the r5 probes: threads_in_calls_max 4 at --cpu-threads 4)",
            "runner.replacing: VERBOSE: Replacing 2376 out of 2432 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 57 partitions for subgraph 0 (main).",
            "line PASS: 124 questions, same winner 119/119 (not near ties), near ties kept 5/5, max |Δp| 0.000908779, mean |Δp| 9.88097e-05, control n/a (the file holds no request of the control pair), non-finite read-outs 0, winner = the Mac CPU's on the same file 119/119",
            "Run memory (29 samples): VmHWM_max_kB 1648784 VmSwap_max_kB 0 MemAvailable_min_kB 8776356 kgsl_page_alloc_max_B "
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "compile_ms": 863.1,
            "line_bar_control_abs_dp": 0.02,
            "line_bar_max_abs_dp": 0.02,
            "line_bar_mean_abs_dp": 0.002,
            "line_bar_near_tie_gap": 0.02,
            "line_mac_same_file_non_near_tie": 119.0,
            "line_max_abs_dp": 0.0009087786688101729,
            "line_mean_abs_dp": 9.880967733262983e-05,
            "line_near_tie": 5.0,
            "line_non_near_tie": 119.0,
            "line_nonfinite_questions": 0.0,
            "line_pass": 1.0,
            "line_questions": 124.0,
            "line_winner_equal_mac_non_near_tie": 119.0,
            "line_winner_equal_near_tie": 5.0,
            "line_winner_equal_non_near_tie": 119.0,
            "threads": 5.0,
            "vmhwm_max_kb": 1648784.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the line: 124 questions of the L768 set against the source model's reference; the C-API runner build 4946d828 (LiteRT 2.3.0), CPU XNNPACK, 5 threads (--cpu-threads 5)",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__pixel-11-pro.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "cpu_xnnpack",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2376,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocketdec-capi-runner",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Google Tensor G6",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "1900.88 ms (min 1723.38, max 2069.74; median of 8 after 2 warm-ups, write+run+readback; run only 1899.06 ms; load 0.84 s… (the cell's value; parity cell: PASS; gate PASS, timing run = check run (5/5 q, dumps equal))",
            "(tag p11p_xt5t_d_L768_rt1, run r7): start 2026-11-08 09:19:47, request F2/customer_service_000006 (586 tokens), write_run_read_ms median 1900.8760000000002 min 1723.379 max 2069.739 n 8, run_only_ms median 1899.0639999999999",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 2/2 OK; extras: rest=180 sample=1 args=--cpu-threads,5",
            "Device: Pixel 11 Pro 17 CD1A.260905.001.B1",
            "clock samples: 22 samples, 0 capped, GPU level max 0",
            "parity: [\"p11p_xt5t_d_L768_rt1\"]: gate PASS, timing run = check run (5/5 questions equal in every digit, max |Δp| between 0.0), against the check run",
            "VERBOSE: Replacing 2376 out of 2432 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 57 partitions for subgraph 0 (main).",
            "the cell's takes (take id: median, capped): 590: 2124 ms capped; 625: 1901 ms; the cell is take 625, the fastest uncapped take",
            "threads: the build 4946d828 passes --cpu-threads to LiteRT"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": 1900.8760000000002,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "capped": 0.0,
            "capped_samples": 0.0,
            "clock_samples": 22.0,
            "compile_ms": 838.2,
            "latency_max_ms": 2069.739,
            "latency_min_ms": 1723.379,
            "request_questions": 5.0,
            "request_tokens": 586.0,
            "run_only_p50_ms": 1899.0639999999999,
            "threads": 5.0,
            "timed_calls": 8.0,
            "vmhwm_max_kb": 1650432.0,
            "warmup_calls": 2.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "timing: the typed-decisions test row `customer_service_000006` (586 tokens, 5 questions), the median of 8 calls after 2 warm-up calls in one process, a call = write the four inputs, run, read `hidden` back; the C-API runner build 4946d828 (LiteRT 2.3.0), CPU XNNPACK, 5 threads (--cpu-threads 5)",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__pixel-11-pro.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "gpu_mldrift",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2432,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocketdec-capi-runner",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Google Tensor G6",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "(tag p11p_gf_L768, run r7): run DONE rc 0, start 2026-11-08 07:06:04, 124 questions",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 2/2 OK",
            "Device: Pixel 11 Pro 17 CD1A.260905.001.B1",
            "runner.replacing: VERBOSE: Replacing 2432 out of 2432 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).",
            "line PASS: 124 questions, same winner 119/119 (not near ties), near ties kept 5/5, max |Δp| 0.000906431, mean |Δp| 9.88447e-05, control n/a (the file holds no request of the control pair), non-finite read-outs 0, winner = the Mac CPU's on the same file 119/119",
            "Run memory (21 samples): VmHWM_max_kB 2503256 VmSwap_max_kB 0 MemAvailable_min_kB 8042476 kgsl_page_alloc_max_B "
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "compile_ms": 32583.9,
            "line_bar_control_abs_dp": 0.02,
            "line_bar_max_abs_dp": 0.02,
            "line_bar_mean_abs_dp": 0.002,
            "line_bar_near_tie_gap": 0.02,
            "line_mac_same_file_non_near_tie": 119.0,
            "line_max_abs_dp": 0.0009064306168956593,
            "line_mean_abs_dp": 9.884466320519424e-05,
            "line_near_tie": 5.0,
            "line_non_near_tie": 119.0,
            "line_nonfinite_questions": 0.0,
            "line_pass": 1.0,
            "line_questions": 124.0,
            "line_winner_equal_mac_non_near_tie": 119.0,
            "line_winner_equal_near_tie": 5.0,
            "line_winner_equal_non_near_tie": 119.0,
            "vmhwm_max_kb": 2503256.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the line: 124 questions of the L768 set against the source model's reference; the C-API runner build 4946d828 (LiteRT 2.3.0), GPU OpenCL, GpuOptions precision FP32",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__pixel-11-pro.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "gpu_mldrift",
          "context_length": null,
          "date": "2026-11-08",
          "decode_tokens_per_s": null,
          "delegated_ops": 2432,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-mnnpocketdec-capi-runner",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert",
            "runtime_version": "2.3.0",
            "soc": "Google Tensor G6",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "716.41 ms (min 702.14, max 730.20; median of 8 after 2 warm-ups, write+run+readback; run only 2.38 ms; load 15.56 s; VmH… (the cell's value; parity cell: PASS; gate PASS, timing run = check run (5/5 q, dumps equal))",
            "(tag p11p_gfs_d_L768_ab2, run r7): start 2026-11-08 07:24:51, request F2/customer_service_000006 (586 tokens), write_run_read_ms median 716.4125 min 702.139 max 730.195 n 8, run_only_ms median 2.3760000000000003",
            "Push mnn_pocket_dec_ns4_in_6-27_post_6-27_final_L768_w16.tflite rc=0 bytes 488880528/488880528 sha256 phone=7dc7ca0596129e7dc0eec2e952b0b07bacbbd048d17d7988b3f13c4903c10034 = mac = the Hub LFS sha256",
            "Runner 4946d828cc15b77b… and libs: 2/2 OK; extras: rest=180 sample=1",
            "Device: Pixel 11 Pro 17 CD1A.260905.001.B1",
            "clock samples: 16 samples, 0 capped, GPU level max 0",
            "parity: [\"p11p_gfs_d_L768_ab2\"]: gate PASS, timing run = check run (5/5 questions equal in every digit, max |Δp| between 0.0), against the check run",
            "VERBOSE: Replacing 2432 out of 2432 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).",
            "the cell's takes (take id: median, capped): 587: 720.2 ms; 588: 718.5 ms; 589: 716.4 ms; the cell is take 589, the fastest uncapped take",
            "other rows of this file, request and backend, not takes of this cell: 631: GPU precision fp16acc32; 632: GPU precision fp16acc32",
            "the same request on the L1024 and L768 files one run after the other, L1024, L768, L768, L1024 (abba_* metrics)"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": 716.4125,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "abba_a1_l1024_ms": 1032.3385,
            "abba_a2_l1024_ms": 1033.2525,
            "abba_b1_l768_ms": 718.4795,
            "abba_b2_l768_ms": 716.4125,
            "arm_files_passed": 8.0,
            "arm_files_run": 8.0,
            "capped": 0.0,
            "capped_samples": 0.0,
            "clock_samples": 16.0,
            "compile_ms": 15562.9,
            "latency_max_ms": 730.195,
            "latency_min_ms": 702.139,
            "request_questions": 5.0,
            "request_tokens": 586.0,
            "run_only_p50_ms": 2.3760000000000003,
            "timed_calls": 8.0,
            "vmhwm_max_kb": 2499172.0,
            "warmup_calls": 2.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "timing: the typed-decisions test row `customer_service_000006` (586 tokens, 5 questions), the median of 8 calls after 2 warm-up calls in one process, a call = write the four inputs, run, read `hidden` back; the C-API runner build 4946d828 (LiteRT 2.3.0), GPU OpenCL, GpuOptions precision FP32",
          "total_ops": 2432,
          "ttft_ms": null
        },
        "source": "data/device_runs/2.3.0/2026-11-08/mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w__pixel-11-pro.json"
      }
    ]
  },
  "model": {
    "family": "mnn-pocket",
    "id": "mnn-pocket-0.3b-dec__mnn-pocket-0.3b-dec_l768_fp16w",
    "license": "apache-2.0",
    "source_url": "https://huggingface.co/litert-community/MNN-Pocket-0.3B-Dec-LiteRT",
    "task": "text-classification"
  },
  "pitfalls": [
    "A decision model, not a chat model: it takes a `state` and typed `questions` (`choice`, `noul`, `score`) and returns the probability of every candidate in one forward pass; it never generates text (README L18).",
    "The graph is the model body only: it takes the embeddings of the request's tokens, the rotary tables and the attention mask and returns the hidden states after the final norm; the Python host in `host/` builds the request, takes the embeddings from `embedding.safetensors` (the source model's token table) and reads the answers with the source model's head and temperature, as the source model's `inference.py` does (README L20, L147-L148).",
    "Each file is a static window: `<n>` in the name is the longest request it takes; run the smallest file that holds the request. A request longer than 2,048 tokens raises `ValueError` in the host, which never cuts it; the source model's own limit is 32,768 tokens (README L71, L84).",
    "The checks measure agreement with the source model's `inference.py`, unchanged, on its own files (float32, CPU, one thread): 2,409 questions on the Mac, a fixed subset of 16 to 322 questions per file on the phones; the bar is the same winner on every question whose two most likely options are more than 0.02 apart, a largest |dp| of at most 0.02 and a mean of at most 0.002 (README L88, L102).",
    "The repository has no Android host yet: an Android app has to port `host/pocket_dec_litert.py`; the phones ran LiteRT 2.3.0 through its C API with inputs the Mac computed with the host's code (README L137).",
    "On a GPU set float32 precision (`GpuOptions(enforce_f32=True)` in the Python host, `CompiledModel.GpuOptions.Precision.FP32` in Kotlin, `kLiteRtDelegatePrecisionFp32` in the C API): an app that sets no precision gets FP16 on both phones, and FP16 changes the answers: among questions whose top two options are more than 0.02 apart the winner changed on up to 5 of 305 on the Galaxy S26, 1 of 305 on the Pixel 11 Pro and 9 of 2,314 on Apple Metal at its default precision, and a probability moved by up to 0.068, 0.029 and 0.058 (README L73).",
    "FP16 with FP32 accumulation passes the bar on the Galaxy S26 (44 of 54 near ties keep the reference's winner, largest |dp| 0.0126) and misses the mean on 3 of the 6 files it ran on the Pixel 11 Pro; it can flip a near tie: in the source model's default example (`department`: `shipping` 0.41064 against `returns` 0.40709) it answers `returns` (README L96, L102, L104).",
    "Set the CPU thread count: with no CPU options LiteRT 2.3.0's C API ran on one thread, and the 128-token file took 1.3 times as long as at 4 threads on the Galaxy S26 and 2.2 times as long on the Pixel 11 Pro; the card's phone CPU rows use 4 threads (Galaxy S26) and 6 (Pixel 11 Pro), whose probabilities equal those of one thread to every digit (README L133).",
    "LiteRT's GPU cache (`serializationDir`, `modelCacheKey`, `serializeProgramCache` and `serializeExternalTensors` in `CompiledModel.GpuOptions`) cut the Pixel 11 Pro's creation of the 128-token model from 15.5 s to 1.8 to 2.0 s, but each call then took 12 to 19 % longer; on the Galaxy S26 it cut 1.6 s to 1.3 s at the same speed; the cache takes 0.93 GiB on disk and raises the process's peak by about 0.9 GiB (README L135)."
  ],
  "schema_version": "1.2"
}
