{
  "artifacts": [
    {
      "file": "oc_e_non_v1_int8_ekv4096.litertlm",
      "sha256": "9cba203de52dc34d772660c95b7629536c44916604d84fa45dee43e1cf514b4a",
      "size_mb": 3872.103
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "p-e-w/Oc-e-Non-v1 rev dbf4e269 (bf16) -> export_hf with its GPU composites; int8 per-channel weights for every linear layer and the token table; fp32 GPU activations; LiteRT-LM 0.19.1's Qwen3 configuration with the checkpoint's sampler (top-k 20, top-p 0.8, temperature 0.7), <|endoftext|> as a second stop token and no thought channel; the checkpoint's tokenizer.json byte for byte (card L36, L69-L70)",
    "quantization": "int8 per channel for every linear layer and the token table; fp32 GPU activations (card L36)",
    "tool": "litert-torch 0.10.0 export_hf with its GPU composites, quantized with ai-edge-quantizer 0.10.1; one command rebuilds both files (the recipe in john-rocky/hf-to-litertlm REPRODUCE.md) (card L18, L69, L71)",
    "tool_version": "litert-torch 0.10.0, ai-edge-quantizer 0.10.1; litert_lm_advanced_main built from LiteRT-LM v0.19.0 (771b06e1) for the phone numbers, the PyPI litert-lm 0.19.2 on an Apple M4 Max for the accuracy checks (card L18, L42, L52, L65)"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "cpu",
          "context_length": 2048,
          "date": "2026-11-15",
          "decode_tokens_per_s": 7.4475,
          "delegated_ops": 2926,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-ocenon-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Landed 2026-11-15 from the close-out: mlboydaisuke/Oc-e-Non-v1-LiteRT is public and ungated (license apache-2.0, base_model p-e-w/Oc-e-Non-v1 quantized, library_name litert-lm, pipeline_tag text-generation), upload commit fbcff6e942 (2026-11-15 09:10:45 JST), main 19cd5faf33 (a README-only commit, 2026-11-15 12:29:13 JST) at the read; the tree's LFS oid and size of oc_e_non_v1_int8_ekv4096.litertlm = artifact_sha256 and 4060193936 B (anonymous, read 2026-11-15 12:47:48 JST).",
            "binary: litert_lm_advanced_main built for android_arm64 from the LiteRT-LM v0.19.0 tag (commit 771b06e1, 2026-10-28T23:05:58Z; the kit's BUILD_INFO: describe v0.19.0, NDK 29.0.13113456, bazel 7.6.1, built 2026-10-30T19:14:27Z), sha256 2e0b5097..., with the .so files the same build wrote (the kit's SHA256SUMS; every run's setup checked each file's sha256 on the phone: MATCH); its log reads 'Statically linked GPU accelerator registered.' on every run (asserted)",
            "protocol: a benchmark script (one process per run, a ready check before each: thermal, CPU cap, rest), the run scorer tools/score_leg.py; `./litert_lm_advanced_main --backend=cpu --model_path=<the file> --input_prompt_file=prompts/bench.txt --max_num_tokens=2048 --max_output_tokens=2048 --async=false --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=3 --use_session=false` (each run's .meta.json cmd; the phone-side memory guard 1,000,000 kB)",
            "2 push(es) of ocn_i8, each 'sha256 phone=9cba203de52dc34d... MATCH'",
            "the cell (the card's Performance rule): per process the mean of iterations 2-3 of one engine (iteration 1 = the warm-up), the cell = the mean over the 2 process(es) of this file and backend, prefill, decode and TTFT each; peak_mem_mb = the larger VmHWM of the processes (kB x 1,024 / 10^6; VmHWM does not count the memory the GPU driver holds); each process started after the chain's ready check, 305-306 s after the previous process",
            "the card's Performance row (README L60-L63 at main 19cd5faf, the command line L65): int8 CPU 79.5 / 7.4 tok/s, 13.1 s, 7.16 GB = this cell at the card's digits",
            "r1s-t8-i8-cpu (process 1; started 2026-11-15 08:04:07 JST in r1s hold 3 (11-15 07:59:01-08:22:04 JST); warm (thermal 1, cap none, kgsl_pwrlevel 0); rested 305s since 07:59:02): iterations 112.07 / 8.1 tok/s, TTFT 9.26 s; 75.98 / 7.64 tok/s, TTFT 13.61 s; 86.78 / 7.3 tok/s, TTFT 11.94 s; value (iterations 2-3) 81.38 / 7.47 tok/s, TTFT 12.775 s; init 4834.86 ms; VmHWM_max_kB 6657580 VmSwap_max_kB 1409408 MemAvailable_min_kB 4266604 kgsl_page_alloc_max_B 218071040",
            "CPU = XNNPACK, 'number_of_threads: 4' printed; the delegate lines: prefill_1031 327 / 556 nodes (364 partitions), then on the second pass 2926 / 3226 (146 partitions); decode 290 / 518 (364), then 2708 / 3007 (146); 434 lines for the odml.* composite subgraphs (full_delegation false: the log states a split; the counts are the second pass's prefill_1031 line)",
            "r1s-t9-i8-cpu (process 2; started 2026-11-15 08:11:37 JST in r1s hold 3 (11-15 07:59:01-08:22:04 JST); warm (thermal 1, cap none, kgsl_pwrlevel 0); rested 306s since 08:06:31): iterations 117.84 / 8.1 tok/s, TTFT 8.81 s; 68.62 / 7.6 tok/s, TTFT 15.06 s; 86.56 / 7.25 tok/s, TTFT 11.97 s; value (iterations 2-3) 77.59 / 7.42 tok/s, TTFT 13.515 s; init 7451.75 ms; VmHWM_max_kB 6991612 VmSwap_max_kB 1272064 MemAvailable_min_kB 4548552 kgsl_page_alloc_max_B 271138816",
            "CPU = XNNPACK, 'number_of_threads: 4' printed; the delegate lines: prefill_1031 327 / 556 nodes (364 partitions), then on the second pass 2926 / 3226 (146 partitions); decode 290 / 518 (364), then 2708 / 3007 (146); 434 lines for the odml.* composite subgraphs (full_delegation false: the log states a split; the counts are the second pass's prefill_1031 line)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "cpu_threads": 4.0,
            "decode_tokens": 256.0,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "proc1_init_total_ms": 4834.86,
            "proc1_iter1_decode_tok_s": 8.1,
            "proc1_iter1_prefill_tok_s": 112.07,
            "proc1_iter1_ttft_s": 9.26,
            "proc1_iter2_decode_tok_s": 7.64,
            "proc1_iter2_prefill_tok_s": 75.98,
            "proc1_iter2_ttft_s": 13.61,
            "proc1_iter3_decode_tok_s": 7.3,
            "proc1_iter3_prefill_tok_s": 86.78,
            "proc1_iter3_ttft_s": 11.94,
            "proc1_rest_s_before": 305.0,
            "proc1_thermal_status_at_start": 1.0,
            "proc1_vmhwm_kb": 6657580.0,
            "proc2_init_total_ms": 7451.75,
            "proc2_iter1_decode_tok_s": 8.1,
            "proc2_iter1_prefill_tok_s": 117.84,
            "proc2_iter1_ttft_s": 8.81,
            "proc2_iter2_decode_tok_s": 7.6,
            "proc2_iter2_prefill_tok_s": 68.62,
            "proc2_iter2_ttft_s": 15.06,
            "proc2_iter3_decode_tok_s": 7.25,
            "proc2_iter3_prefill_tok_s": 86.56,
            "proc2_iter3_ttft_s": 11.97,
            "proc2_rest_s_before": 306.0,
            "proc2_thermal_status_at_start": 1.0,
            "proc2_vmhwm_kb": 6991612.0,
            "processes": 2.0
          },
          "output_match": null,
          "peak_mem_mb": 7159.411,
          "prefill_tokens_per_s": 79.485,
          "provenance": "measured",
          "runs": true,
          "total_ops": 3226,
          "ttft_ms": 13145.0
        },
        "source": "data/device_runs/0.19.0/2026-11-15/oc-e-non-v1-int8-ekv4096__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": 2048,
          "date": "2026-11-15",
          "decode_tokens_per_s": 12.77,
          "delegated_ops": 556,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-ocenon-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Landed 2026-11-15 from the close-out: mlboydaisuke/Oc-e-Non-v1-LiteRT is public and ungated (license apache-2.0, base_model p-e-w/Oc-e-Non-v1 quantized, library_name litert-lm, pipeline_tag text-generation), upload commit fbcff6e942 (2026-11-15 09:10:45 JST), main 19cd5faf33 (a README-only commit, 2026-11-15 12:29:13 JST) at the read; the tree's LFS oid and size of oc_e_non_v1_int8_ekv4096.litertlm = artifact_sha256 and 4060193936 B (anonymous, read 2026-11-15 12:47:48 JST).",
            "binary: litert_lm_advanced_main built for android_arm64 from the LiteRT-LM v0.19.0 tag (commit 771b06e1, 2026-10-28T23:05:58Z; the kit's BUILD_INFO: describe v0.19.0, NDK 29.0.13113456, bazel 7.6.1, built 2026-10-30T19:14:27Z), sha256 2e0b5097..., with the .so files the same build wrote (the kit's SHA256SUMS; every run's setup checked each file's sha256 on the phone: MATCH); its log reads 'Statically linked GPU accelerator registered.' on every run (asserted)",
            "protocol: a benchmark script (one process per run, a ready check before each: thermal, CPU cap, rest), the run scorer tools/score_leg.py; `./litert_lm_advanced_main --backend=gpu --model_path=<the file> --input_prompt_file=prompts/bench.txt --max_num_tokens=2048 --max_output_tokens=2048 --async=false --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=3 --use_session=false` (each run's .meta.json cmd; the phone-side memory guard 1,000,000 kB)",
            "2 push(es) of ocn_i8, each 'sha256 phone=9cba203de52dc34d... MATCH'",
            "the cell (the card's Performance rule): per process the mean of iterations 2-3 of one engine (iteration 1 = the warm-up), the cell = the mean over the 2 process(es) of this file and backend, prefill, decode and TTFT each; peak_mem_mb = the larger VmHWM of the processes (kB x 1,024 / 10^6; VmHWM does not count the memory the GPU driver holds); each process started after the chain's ready check, 244-307 s after the previous process",
            "the card's Performance row (README L60-L63 at main 19cd5faf, the command line L65): int8 GPU 119.5 / 12.8 tok/s, 8.7 s, 1.14 GB = this cell at the card's digits",
            "r1s-t3-i8-gpu (process 1; started 2026-11-15 02:56:51 JST in r1s hold 1 (11-15 02:38:47-03:04:08 JST); warm (thermal 1, cap none, kgsl_pwrlevel 0); rested 307s since 02:51:43): iterations 163.06 / 12.8 tok/s, TTFT 6.36 s; 127.65 / 12.82 tok/s, TTFT 8.1 s; 113.27 / 12.63 tok/s, TTFT 9.12 s; value (iterations 2-3) 120.46 / 12.73 tok/s, TTFT 8.61 s; init 8202.45 ms; VmHWM_max_kB 1110748 VmSwap_max_kB 221568 MemAvailable_min_kB 1547180 kgsl_page_alloc_max_B 6197055488",
            "r1s-t3-i8-gpu /: prefill_1031: 556 / 556 on LITERT_CL, 1 partition; decode: 518 / 518 on LITERT_CL, 1 partition",
            "r1s-t4-i8-gpu (process 2; started 2026-11-15 03:02:30 JST in r1s hold 1 (11-15 02:38:47-03:04:08 JST); warm-deadline (thermal 1, cap none, kgsl_pwrlevel 0); rested 244s since 02:58:26): iterations 162.56 / 13.0 tok/s, TTFT 6.38 s; 130.87 / 12.93 tok/s, TTFT 7.9 s; 106.22 / 12.7 tok/s, TTFT 9.72 s; value (iterations 2-3) 118.55 / 12.81 tok/s, TTFT 8.81 s; init 7770.82 ms; VmHWM_max_kB 1105456 VmSwap_max_kB 217984 MemAvailable_min_kB 1761324 kgsl_page_alloc_max_B 6195523584",
            "r1s-t4-i8-gpu /: prefill_1031: 556 / 556 on LITERT_CL, 1 partition; decode: 518 / 518 on LITERT_CL, 1 partition"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "proc1_init_total_ms": 8202.45,
            "proc1_iter1_decode_tok_s": 12.8,
            "proc1_iter1_prefill_tok_s": 163.06,
            "proc1_iter1_ttft_s": 6.36,
            "proc1_iter2_decode_tok_s": 12.82,
            "proc1_iter2_prefill_tok_s": 127.65,
            "proc1_iter2_ttft_s": 8.1,
            "proc1_iter3_decode_tok_s": 12.63,
            "proc1_iter3_prefill_tok_s": 113.27,
            "proc1_iter3_ttft_s": 9.12,
            "proc1_rest_s_before": 307.0,
            "proc1_thermal_status_at_start": 1.0,
            "proc1_vmhwm_kb": 1110748.0,
            "proc2_init_total_ms": 7770.82,
            "proc2_iter1_decode_tok_s": 13.0,
            "proc2_iter1_prefill_tok_s": 162.56,
            "proc2_iter1_ttft_s": 6.38,
            "proc2_iter2_decode_tok_s": 12.93,
            "proc2_iter2_prefill_tok_s": 130.87,
            "proc2_iter2_ttft_s": 7.9,
            "proc2_iter3_decode_tok_s": 12.7,
            "proc2_iter3_prefill_tok_s": 106.22,
            "proc2_iter3_ttft_s": 9.72,
            "proc2_rest_s_before": 244.0,
            "proc2_thermal_status_at_start": 1.0,
            "proc2_vmhwm_kb": 1105456.0,
            "processes": 2.0
          },
          "output_match": null,
          "peak_mem_mb": 1137.406,
          "prefill_tokens_per_s": 119.5025,
          "provenance": "measured",
          "runs": true,
          "total_ops": 556,
          "ttft_ms": 8710.0
        },
        "source": "data/device_runs/0.19.0/2026-11-15/oc-e-non-v1-int8-ekv4096__galaxy-s26.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-11-14",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.2",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Landed 2026-11-15 from the close-out: mlboydaisuke/Oc-e-Non-v1-LiteRT is public and ungated (license apache-2.0, base_model p-e-w/Oc-e-Non-v1 quantized, library_name litert-lm, pipeline_tag text-generation), upload commit fbcff6e942 (2026-11-15 09:10:45 JST), main 19cd5faf33 (a README-only commit, 2026-11-15 12:29:13 JST) at the read; the tree's LFS oid and size of oc_e_non_v1_int8_ekv4096.litertlm = artifact_sha256 and 4060193936 B (anonymous, read 2026-11-15 12:47:48 JST).",
            "runtime: the PyPI litert-lm-api 0.19.2 macOS arm64 wheel (Python 3.13.13; liblitert-lm.dylib sha256 13421998..., 50,610,352 B), its Python API (tools/lt_run.py): one Engine per file and backend with max_num_tokens 4096 and no cache directory, a new conversation per question, greedy = SamplerConfig(top_k=1, top_p=0.95, temperature=1.0, seed=0), the file's own activation declaration; CPU = XNNPACK (the default thread count), GPU = WebGPU on Metal; a process a window paused for more than 10 minutes was not used past the pause (send_message's deadline): its unanswered questions went to a new process. Paths are relative",
            "output_match = line 3 as written before any output (2026-11-14 16:33 JST): (a) the file's verdict equals the reference's on at least 11 of 12, (b) every question the reference ends before the cap ends before the cap here, (c) turn 3 recalls 37 and no turn reaches the cap; reference = Transformers 5.22.1 with the bf16 weights computed in float32 on the Mac CPU, the checkpoint's chat template, greedy",
            "(started 2026-11-14 22:45:01 JST): 12 / 12 verdicts equal (misses: none), no reply at the cap; the conversation: 'Hello, Kaito. 37.' / 'Canberra' / '37'; judge pass"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "conversation_turns": 3.0,
            "conversation_turns_right": 3.0,
            "engine_create_s": 7.5,
            "max_new_tokens": 1024.0,
            "questions": 12.0,
            "verdicts_equal_reference": 12.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the 12 check questions of line 3 (17 + 25, the capital of Japan, the opposite of 'hot', the days of a week, 'thank you' in French, 8 times 7, 0.9 or 0.11, the rhyme, a word problem, fib(10) in Python, the planets in order, a Japanese sentence in English), each with ' Answer briefly.' in a new conversation, and a 3-turn conversation, greedy, at most 1,024 new tokens per reply; verdicts against Transformers (float32, greedy)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.2/2026-11-14/oc-e-non-v1-int8-ekv4096__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-11-14",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.2",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Landed 2026-11-15 from the close-out: mlboydaisuke/Oc-e-Non-v1-LiteRT is public and ungated (license apache-2.0, base_model p-e-w/Oc-e-Non-v1 quantized, library_name litert-lm, pipeline_tag text-generation), upload commit fbcff6e942 (2026-11-15 09:10:45 JST), main 19cd5faf33 (a README-only commit, 2026-11-15 12:29:13 JST) at the read; the tree's LFS oid and size of oc_e_non_v1_int8_ekv4096.litertlm = artifact_sha256 and 4060193936 B (anonymous, read 2026-11-15 12:47:48 JST).",
            "runtime: the PyPI litert-lm-api 0.19.2 macOS arm64 wheel (Python 3.13.13; liblitert-lm.dylib sha256 13421998..., 50,610,352 B), its Python API (tools/lt_run.py): one Engine per file and backend with max_num_tokens 4096 and no cache directory, a new conversation per question, greedy = SamplerConfig(top_k=1, top_p=0.95, temperature=1.0, seed=0), the file's own activation declaration; CPU = XNNPACK (the default thread count), GPU = WebGPU on Metal; a process a window paused for more than 10 minutes was not used past the pause (send_message's deadline): its unanswered questions went to a new process. Paths are relative",
            "output_match = line Q (written before any output): pass = b - c <= 2 sqrt(b + c) against the reference's 100 answers (Transformers, float32, batches of 8, left padding); true when every process passes",
            "(started 2026-11-14 22:50:56 JST): 84 / 100 right, reference 83; b 4 / c 5, bar 6.0 = pass; 3 replies at the 512-token cap (the reference 4)"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "b": 4.0,
            "bar": 6.0,
            "c": 5.0,
            "capped": 3.0,
            "correct": 84.0,
            "max_new_tokens": 512.0,
            "processes": 1.0,
            "questions": 100.0,
            "reference_correct": 83.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "GSM8K test, the first 100 questions with a chain-of-thought prompt, one conversation each, greedy, at most 512 new tokens; paired against Transformers (float32, greedy): b = right in the reference and wrong in the file, c = the reverse, pass = b - c <= 2 sqrt(b + c)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.2/2026-11-14/oc-e-non-v1-int8-ekv4096__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-11-14",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.2",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Landed 2026-11-15 from the close-out: mlboydaisuke/Oc-e-Non-v1-LiteRT is public and ungated (license apache-2.0, base_model p-e-w/Oc-e-Non-v1 quantized, library_name litert-lm, pipeline_tag text-generation), upload commit fbcff6e942 (2026-11-15 09:10:45 JST), main 19cd5faf33 (a README-only commit, 2026-11-15 12:29:13 JST) at the read; the tree's LFS oid and size of oc_e_non_v1_int8_ekv4096.litertlm = artifact_sha256 and 4060193936 B (anonymous, read 2026-11-15 12:47:48 JST).",
            "runtime: the PyPI litert-lm-api 0.19.2 macOS arm64 wheel (Python 3.13.13; liblitert-lm.dylib sha256 13421998..., 50,610,352 B), its Python API (tools/lt_run.py): one Engine per file and backend with max_num_tokens 4096 and no cache directory, a new conversation per question, greedy = SamplerConfig(top_k=1, top_p=0.95, temperature=1.0, seed=0), the file's own activation declaration; CPU = XNNPACK (the default thread count), GPU = WebGPU on Metal; a process a window paused for more than 10 minutes was not used past the pause (send_message's deadline): its unanswered questions went to a new process. Paths are relative",
            "output_match = line 3 as written before any output (2026-11-14 16:33 JST): (a) the file's verdict equals the reference's on at least 11 of 12, (b) every question the reference ends before the cap ends before the cap here, (c) turn 3 recalls 37 and no turn reaches the cap; reference = Transformers 5.22.1 with the bf16 weights computed in float32 on the Mac CPU, the checkpoint's chat template, greedy",
            "(started 2026-11-14 22:12:53 JST): 12 / 12 verdicts equal (misses: none), no reply at the cap; the conversation: 'Kaito, 37' / 'Canberra' / '37'; judge pass"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "conversation_turns": 3.0,
            "conversation_turns_right": 3.0,
            "engine_create_s": 3.6,
            "max_new_tokens": 1024.0,
            "questions": 12.0,
            "verdicts_equal_reference": 12.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the 12 check questions of line 3 (17 + 25, the capital of Japan, the opposite of 'hot', the days of a week, 'thank you' in French, 8 times 7, 0.9 or 0.11, the rhyme, a word problem, fib(10) in Python, the planets in order, a Japanese sentence in English), each with ' Answer briefly.' in a new conversation, and a 3-turn conversation, greedy, at most 1,024 new tokens per reply; verdicts against Transformers (float32, greedy)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.2/2026-11-14/oc-e-non-v1-int8-ekv4096__mac-studio-m4-max.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-11-14",
          "decode_tokens_per_s": null,
          "delegated_ops": 2926,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-ocenon-devicebench",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Google Tensor G6",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Landed 2026-11-15 from the close-out: mlboydaisuke/Oc-e-Non-v1-LiteRT is public and ungated (license apache-2.0, base_model p-e-w/Oc-e-Non-v1 quantized, library_name litert-lm, pipeline_tag text-generation), upload commit fbcff6e942 (2026-11-15 09:10:45 JST), main 19cd5faf33 (a README-only commit, 2026-11-15 12:29:13 JST) at the read; the tree's LFS oid and size of oc_e_non_v1_int8_ekv4096.litertlm = artifact_sha256 and 4060193936 B (anonymous, read 2026-11-15 12:47:48 JST).",
            "binary: litert_lm_advanced_main built for android_arm64 from the LiteRT-LM v0.19.0 tag (commit 771b06e1, 2026-10-28T23:05:58Z; the kit's BUILD_INFO: describe v0.19.0, NDK 29.0.13113456, bazel 7.6.1, built 2026-10-30T19:14:27Z), sha256 2e0b5097..., with the .so files the same build wrote (the kit's SHA256SUMS; every run's setup checked each file's sha256 on the phone: MATCH); its log reads 'Statically linked GPU accelerator registered.' on every run (asserted)",
            "1 push(es) of ocn_i8, each 'sha256 phone=9cba203de52dc34d... MATCH'",
            "output_match = line D (written 2026-11-14 16:33 JST before any output): the same verdict as the reference on at least 7 of the 8 questions and no reply at the cap (the work applied the same rule to the CPU runs); the reference (Transformers 5.22.1, the bf16 weights computed in float32 on the Mac CPU, greedy) answers all 8 right",
            "(r1p, r1p hold 1 (11-14 22:59:08-23:22:24 JST); first run 2026-11-14 23:20:38 JST, last 23:22:19): q1 '42' PASS; q2 'Tokyo' PASS; q3 'cold' PASS; q4 '7' PASS; q5 'Merci' PASS; q6 '56' PASS; q7 '0.9' PASS; q8 'blue' PASS; reply tokens [3, 3, 2, 2, 3, 3, 4, 2]",
            "CPU = XNNPACK, 'number_of_threads: 4' printed; the delegate lines: prefill_1031 327 / 556 nodes (364 partitions), then on the second pass 2926 / 3226 (146 partitions); decode 290 / 518 (364), then 2708 / 3007 (146); 434 lines for the odml.* composite subgraphs (full_delegation false: the log states a split; the counts are the second pass's prefill_1031 line)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "answers_right": 8.0,
            "max_num_tokens": 1024.0,
            "max_output_tokens": 256.0,
            "questions": 8.0,
            "reply_tokens_max": 4.0,
            "verdicts_equal_reference": 8.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the first 8 of the 12 check questions of line 3 (17 + 25, the capital of Japan, the opposite of 'hot', the days of a week, 'thank you' in French, 8 times 7, 0.9 or 0.11, the rhyme), each with ' Answer briefly.' in its own process, --max_num_tokens=1024 --max_output_tokens=256, no sampler flags (the file's sampler); each verdict by the question's regex against Transformers (float32, greedy)",
          "total_ops": 3226,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.0/2026-11-14/oc-e-non-v1-int8-ekv4096__pixel-11-pro.json"
      },
      {
        "device": "pixel-11-pro",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-11-14",
          "decode_tokens_per_s": null,
          "delegated_ops": 556,
          "env": {
            "device": "Pixel 11 Pro",
            "machine_label": "pixel-11-pro-ocenon-devicebench",
            "os_build": "Android 17 (CD1A.260905.001.B1)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Google Tensor G6",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "Landed 2026-11-15 from the close-out: mlboydaisuke/Oc-e-Non-v1-LiteRT is public and ungated (license apache-2.0, base_model p-e-w/Oc-e-Non-v1 quantized, library_name litert-lm, pipeline_tag text-generation), upload commit fbcff6e942 (2026-11-15 09:10:45 JST), main 19cd5faf33 (a README-only commit, 2026-11-15 12:29:13 JST) at the read; the tree's LFS oid and size of oc_e_non_v1_int8_ekv4096.litertlm = artifact_sha256 and 4060193936 B (anonymous, read 2026-11-15 12:47:48 JST).",
            "binary: litert_lm_advanced_main built for android_arm64 from the LiteRT-LM v0.19.0 tag (commit 771b06e1, 2026-10-28T23:05:58Z; the kit's BUILD_INFO: describe v0.19.0, NDK 29.0.13113456, bazel 7.6.1, built 2026-10-30T19:14:27Z), sha256 2e0b5097..., with the .so files the same build wrote (the kit's SHA256SUMS; every run's setup checked each file's sha256 on the phone: MATCH); its log reads 'Statically linked GPU accelerator registered.' on every run (asserted)",
            "1 push(es) of ocn_i8, each 'sha256 phone=9cba203de52dc34d... MATCH'",
            "output_match = line D (written 2026-11-14 16:33 JST before any output): the same verdict as the reference on at least 7 of the 8 questions and no reply at the cap (the work applied the same rule to the CPU runs); the reference (Transformers 5.22.1, the bf16 weights computed in float32 on the Mac CPU, greedy) answers all 8 right",
            "(r1p, r1p hold 1 (11-14 22:59:08-23:22:24 JST); first run 2026-11-14 23:11:20 JST, last 23:18:37): q1 '42' PASS; q2 'Tokyo' PASS; q3 'Cold' PASS; q4 '7 days' PASS; q5 'Merci' PASS; q6 '56' PASS; q7 '0.9' PASS; q8 'blue' PASS; reply tokens [3, 3, 2, 3, 3, 3, 4, 2]",
            "r1p-i8-gpu-q1 /: prefill_1031: 556 / 556 on LITERT_CL, 1 partition; decode: 518 / 518 on LITERT_CL, 1 partition"
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "answers_right": 8.0,
            "max_num_tokens": 1024.0,
            "max_output_tokens": 256.0,
            "questions": 8.0,
            "reply_tokens_max": 4.0,
            "verdicts_equal_reference": 8.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the first 8 of the 12 check questions of line 3 (17 + 25, the capital of Japan, the opposite of 'hot', the days of a week, 'thank you' in French, 8 times 7, 0.9 or 0.11, the rhyme), each with ' Answer briefly.' in its own process, --max_num_tokens=1024 --max_output_tokens=256, no sampler flags (the file's sampler); each verdict by the question's regex against Transformers (float32, greedy)",
          "total_ops": 556,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.0/2026-11-14/oc-e-non-v1-int8-ekv4096__pixel-11-pro.json"
      }
    ]
  },
  "model": {
    "family": "qwen3",
    "id": "oc-e-non-v1-int8-ekv4096",
    "license": "apache-2.0",
    "source_url": "https://huggingface.co/mlboydaisuke/Oc-e-Non-v1-LiteRT",
    "task": "text-generation"
  },
  "pitfalls": [
    "Use LiteRT-LM 0.19.0 or newer: the files were checked on 0.19.0 (Android) and 0.19.2 (Mac); earlier releases were not tried, nor phones with 8 GB of memory (card L25).",
    "An incomplete download: if `run` stops right after the download with no error message from the CLI (a bus error; exit status 138 on a Mac) or with an error log that contains `too large for file size`, the litert-lm CLI (tested on 0.19.2 and litert-lm-nightly 0.20.0.dev20261112; 0.14.0 to 0.19.1 have the same download code) can keep an incomplete download in ~/.litert-lm/cache/huggingface/mlboydaisuke/Oc-e-Non-v1-LiteRT/ and use it again without checking its size; delete that file and run the command again (card L27).",
    "The file for answers closer to the original model, not for speed: the same verdict on 12 of 12 Mac questions (the Mixed INT4 file 11 of 12); on a Galaxy S26 GPU it decodes 12.8 tokens/s, the Mixed INT4 file 1.4 times as fast (card L35-L36, L60-L61).",
    "4,096 tokens of context: the prompt, the reply and earlier turns together (card L38); the card's Galaxy S26 rows ran at --max_num_tokens=2048 (card L65).",
    "The Galaxy S26 peak (VmHWM) does not count the memory the GPU driver holds (card L65).",
    "LiteRT-LM .litertlm bundle: LiteRT.js cannot run it, so delegation stays null and there is no browser block; device rows come from data/device_runs/."
  ],
  "schema_version": "1.2"
}
