{
  "artifacts": [
    {
      "file": "granite-4.1-8b_int4.litertlm",
      "sha256": "7de1d77fb951572d12b8b17984d6418909b841e582d1df7866c5d08b3920ea60",
      "size_mb": 4913.317
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "convert_granite41_8b.sh (hf-to-litertlm): export (6 prefill signatures 1024..1 + decode, KV cache 4096, embedder externalized, no start token, IBM's chat_template.jinja embedded) with the int4 recipe -> int4 block-scale scan -> metadata edit (stop id 100257 only, max_num_tokens 4096) -> tools/gpu_graph/gpu_graph_retrofit.py --decomp exporter --mask add_bcast",
    "quantization": "int4 blockwise-32 + OCTAV on linears, int8 embedding; the embedding table as a separate section (card L25)",
    "tool": "litert-torch (lightweight export, 1e8-byte lazy-constant threshold) + hf-to-litertlm tools/gpu_graph/ GPU-composite rewrite; repro = hf-to-litertlm granite41_work/",
    "tool_version": "litert-torch 0.9.4 (litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1, torch 2.13.0); packed by the litert-lm 0.18.0 CLI"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": 5.3475,
          "delegated_ops": 1860,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-aab2",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "prebuilt-adac974b",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "binary: Google's prebuilt litert_lm_advanced_main + 6 .so (gs://litert/binaries/latest/android_arm64/litert_lm/, object dated 2026-09-18, sha256 adac974b…)",
            "run 1, AB (rewritten graph first): [15:09:34] bundle on device: 7de1d77fb951572d12b8b17984d6418909b841e582d1df7866c5d08b3920ea60 (5151986096 B, push 140.5 s) = Mac",
            "run 1 prompt process: \"The capital of France is Paris.\"",
            "run 1 bench args: --backend=gpu --model_path=/data/local/tmp/aab2_g41_8b/granite-4.1-8b_int4_gpuc.litertlm --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true --metric_proto_file_path=/data/local/tmp/aab2_g41_8b/metrics.pb",
            "run 2, BA (rewritten graph second): [18:36:19] bundle on device: 7de1d77fb951572d12b8b17984d6418909b841e582d1df7866c5d08b3920ea60 (5151986096 B, push 140.9 s) = Mac",
            "run 2 prompt process: \"The capital of France is Paris.\"",
            "run 2 bench args: --backend=gpu --model_path=/data/local/tmp/aab2_g41_8b/granite-4.1-8b_int4_gpuc.litertlm --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true --metric_proto_file_path=/data/local/tmp/aab2_g41_8b/metrics.pb",
            "VERBOSE: Replacing 1860 out of 1860 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (prefill_1024).",
            "VERBOSE: Replacing 1860 out of 1860 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 1 (prefill_256).",
            "VERBOSE: Replacing 1860 out of 1860 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 2 (prefill_64).",
            "VERBOSE: Replacing 1860 out of 1860 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 3 (prefill_16).",
            "VERBOSE: Replacing 1860 out of 1860 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 4 (prefill_4).",
            "VERBOSE: Replacing 1661 out of 1661 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 5 (prefill_1).",
            "VERBOSE: Replacing 1693 out of 1693 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 6 (decode).",
            "VERBOSE: Replacing 1 out of 4 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 2 partitions for subgraph 0 (decode_embedder)."
          ],
          "failure_class": null,
          "full_delegation": true,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_boot_epoch_s": 1791938976.67,
            "run1_capped_samples": 80.0,
            "run1_cold_init_ms": 30151.35,
            "run1_decode_tok_s": 5.395,
            "run1_decode_tok_s_it1": 8.67,
            "run1_decode_tok_s_it2": 5.92,
            "run1_decode_tok_s_it3": 4.87,
            "run1_init_ms": 24897.63,
            "run1_kgsl_ceiling_min_mhz": 342.0,
            "run1_kgsl_temp_max_c": 90.7,
            "run1_memavail_min_kib": 637384.0,
            "run1_peak_mem_mb": 2038.31,
            "run1_prefill_tok_s": 91.43,
            "run1_prefill_tok_s_it1": 169.33,
            "run1_prefill_tok_s_it2": 106.24,
            "run1_prefill_tok_s_it3": 76.62,
            "run1_sampler_samples": 84.0,
            "run1_skin_c_at_bench_start": 45.0,
            "run1_thermal_status_at_bench_start": 3.0,
            "run1_ttft_s": 11.690000000000001,
            "run2_boot_epoch_s": 1791938976.73,
            "run2_capped_samples": 83.0,
            "run2_cold_init_ms": 25974.99,
            "run2_decode_tok_s": 5.300000000000001,
            "run2_decode_tok_s_it1": 8.58,
            "run2_decode_tok_s_it2": 5.86,
            "run2_decode_tok_s_it3": 4.74,
            "run2_init_ms": 24299.02,
            "run2_kgsl_ceiling_min_mhz": 342.0,
            "run2_kgsl_temp_max_c": 90.7,
            "run2_memavail_min_kib": 630292.0,
            "run2_peak_mem_mb": 2136.36,
            "run2_prefill_tok_s": 90.42500000000001,
            "run2_prefill_tok_s_it1": 168.68,
            "run2_prefill_tok_s_it2": 104.59,
            "run2_prefill_tok_s_it3": 76.26,
            "run2_sampler_samples": 85.0,
            "run2_skin_c_at_bench_start": 45.0,
            "run2_thermal_status_at_bench_start": 3.0,
            "run2_ttft_s": 11.8,
            "runs": 2.0,
            "t_1024_256_s": 59.135
          },
          "output_match": null,
          "peak_mem_mb": 2087.335,
          "prefill_tokens_per_s": 90.9275,
          "provenance": "measured",
          "runs": true,
          "total_ops": 1860,
          "ttft_ms": 11745.0
        },
        "source": "data/device_runs/prebuilt-adac974b/2026-10-15/granite-4.1-8b-int4__galaxy-s26.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-10-14",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.17.1",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (tools/v0171_run.sh, \"$L\" = a symlink to granite-4.1-8b_int4_gpuc.litertlm): uvx --from litert-lm==0.17.1 litert-lm run \"$L\" --backend cpu --prompt \"What is the capital of France? Answer in one word.\" --cache disk ) < /dev/null > \"$o\" 2>&1",
            "stdout: Paris",
            "EXIT=0",
            "== 13:42:21 guard start v0171r_int4_gpuc pids 80866; swap used 1472.38 MB; PhysMem: 40G used (5802M wired, 2773M compressor), 87G unused.; reclaimable 113.69 GB",
            "process footprint (swap guard, 15 s samples, binary GB): max 0 GB"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "answers_with_expect": 1.0,
            "guard_footprint_max_gb": 0.0,
            "processes": 1.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.17.1/2026-10-14/granite-4.1-8b-int4__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": 11.68,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes): litert-lm benchmark granite-4.1-8b_int4.litertlm --backend cpu -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no",
            "pass 1: prefill 52.07 / decode 11.9 tok/s, TTFT 19.7488 s, init 17.8617 s",
            "pass 2: prefill 50.93 / decode 11.46 tok/s, TTFT 20.2036 s, init 18.5157 s",
            "process footprint (swap guard, 15 s samples, binary GB); swap unchanged"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "guard_footprint_max_gb": 32.0,
            "init_s": 18.1887,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 11.9,
            "pass1_init_s": 17.8617,
            "pass1_prefill_tok_s": 52.07,
            "pass1_ttft_s": 19.7488,
            "pass2_decode_tok_s": 11.46,
            "pass2_init_s": 18.5157,
            "pass2_prefill_tok_s": 50.93,
            "pass2_ttft_s": 20.2036,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 41.8115
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 51.5,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 19976.2
        },
        "source": "data/device_runs/0.18.0/2026-10-15/granite-4.1-8b-int4__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": 58.305,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes): litert-lm benchmark granite-4.1-8b_int4.litertlm --backend gpu -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no",
            "pass 1: prefill 722.49 / decode 58.52 tok/s, TTFT 1.4344 s, init 7.4654 s",
            "pass 2: prefill 716.64 / decode 58.09 tok/s, TTFT 1.4461 s, init 7.478 s"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_s": 7.4717,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 58.52,
            "pass1_init_s": 7.4654,
            "pass1_prefill_tok_s": 722.49,
            "pass1_ttft_s": 1.4344,
            "pass2_decode_tok_s": 58.09,
            "pass2_init_s": 7.478,
            "pass2_prefill_tok_s": 716.64,
            "pass2_ttft_s": 1.4461,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 5.8139
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 719.565,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 1440.2
        },
        "source": "data/device_runs/0.18.0/2026-10-15/granite-4.1-8b-int4__mac-studio-m4-max.json"
      },
      {
        "device": "pixel-8a",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-14",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Pixel 8a",
            "machine_label": "pixel-8a-aab2",
            "os_build": "Android 16 (CP1A.260505.005)",
            "runtime": "litert-lm",
            "runtime_version": "prebuilt-adac974b",
            "soc": "Tensor G3",
            "vendor_sdk": null
          },
          "error": "GUARD_KILL avail=327604 pid=13150",
          "evidence": [
            "binary: Google's prebuilt litert_lm_advanced_main + 6 .so (gs://litert/binaries/latest/android_arm64/litert_lm/, object dated 2026-09-18, sha256 adac974b…)",
            "[11:25:21] bundle on device: 7de1d77fb951572d12b8b17984d6418909b841e582d1df7866c5d08b3920ea60 (5151986096 B, push 115.0 s) = Mac",
            "one attempt by rule (--backend=gpu, text prompt 'What is the capital of France? Answer in one sentence.'); a 2-second sampler stops the process when MemAvailable falls under 400,000 kB",
            "VERBOSE: Replacing 1860 out of 1860 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (prefill_1024).",
            "W0000 00:00:1791944725.928400   13150 gpu_model_builder.cc:4455] Weights preparation on Gpu is disabled for PowerVR, Broadcom, Mali GPUs and non-OpenCL/WebGPU/Metal backends.",
            "[11:26:21] prompt_1 done: rc(adb)=0 EXIT=143 wall 55.3 s",
            "GUARD_KILL avail=327604 pid=13150"
          ],
          "failure_class": "memory_cap_exceeded",
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": false,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "memavail_at_start_kib": 4105436.0,
            "memavail_min_kib": 327604.0,
            "sampler_samples": 22.0,
            "seconds_to_stop": 55.3,
            "signatures_compiled": 1.0,
            "vmhwm_kib": 510812.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": false,
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/prebuilt-adac974b/2026-10-14/granite-4.1-8b-int4__pixel-8a.json"
      }
    ]
  },
  "model": {
    "family": "granite",
    "id": "granite-4.1-8b-int4",
    "license": "apache-2.0",
    "source_url": "https://huggingface.co/litert-community/granite-4.1-8b",
    "task": "text-generation"
  },
  "pitfalls": [
    "Runtime: tested on litert-lm 0.18.0 (Mac); on litert-lm 0.17.1 the file answered a plain `run` with \"Paris\" (one CPU process) (card L18).",
    "GPU-composite graph: each layer's two DYNAMIC_UPDATE_SLICE KV writes are one odml.cache_update and the attention products odml.runtime_bmm; on the Mac GPU the rewrite took the 1024 / 256 workload from 6.93 to 5.78 s in the same measurement windows, on the Galaxy S26 GPU the rewrite's decode gain is 1.4x, and before the rewrite the same weights scored the same on GSM8K (93), no question changing between right and wrong (card L54, L90, L98).",
    "Lightweight export: the default litert-torch export path reached 98.8 GB on a 128 GB Mac in the converter passes; with the 1 MiB default lazy-constant threshold the converter left the 40 o_proj weights unquantized behind a constant-only multiply by Granite's residual multiplier 0.22, so the export uses a 1e8-byte threshold (every FULLY_CONNECTED weight an int4 constant, no op with all-constant inputs); the int4 block-scale scan found none of 261,881,856 scales zero or subnormal. Export peaks 50-56 GiB (card L97-L98).",
    "Chat template: IBM's chat_template.jinja byte for byte, no default system message (no system turn without one); no start token; the only stop token <|end_of_text|> (id 100257); no thinking declared. Pass a tool result as the tool message's text: as a Python-API content part it renders empty (card L68).",
    "Conversation preface with tools and no messages: LiteRT-LM renders the chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with 'Failed to apply template: undefined value' when the preface is prefilled at creation (prefillPrefaceOnInit, default false; creating the conversation fails) and in the preface render (renderPrefaceIntoString and its C++ and C API forms); the same with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2); read in the source of LiteRT-LM 0.17.0, 0.18.1 and main, measured with the 0.18.1 PyPI package through its C API on these files' template text; the default path, where the first message is rendered with the preface, is not affected (the 0.18.1 Python API and CLI expose neither the flag nor the preface render); a system message or any initial message in the conversation config avoids it (card L102).",
    "Tool calls come back as text: the model writes <tool_call>{\"name\": ..., \"arguments\": ...}</tool_call> and the app has to parse it (the bundle's generic model type has no tool-call parser) (card L70).",
    "Galaxy S26 GPU figures are warm-phone figures: both benchmark runs started at thermal status 3 (skin 45 C) and the GPU clock ceiling fell from 1300 to 342 MHz inside the runs (80 of 84 and 83 of 85 clock samples below 1300 MHz) (card L90).",
    "Pixel 8a: the one GPU try was stopped by the memory rule (MemAvailable 327,604 kB, 55 s after the start, after prefill_1024 had been delegated 1860/1860 to LITERT_CL; the runtime prepares weights on the CPU side for Mali GPUs). No CPU row on either phone (batch-1 CPU peak factors put the file above both phones' RAM) (card L92).",
    "iOS not measured: the main weights section is 4,731,041,056 bytes, over 2 GiB (card L36).",
    "Accuracy: GSM8K n=100 (greedy, at most 1024 output tokens, fresh process per question, Mac GPU): bf16 96 / int4 93 — three questions differ, all right only in bf16 (card L46-L54).",
    "Mac backend choice: on the Mac CPU the int8 file is faster (its prefill is twice int4's); on both GPUs measured the int4 file is faster (card L36)."
  ],
  "schema_version": "1.2"
}
