{
  "artifacts": [
    {
      "file": "NVIDIA-Nemotron-Nano-9B-v2_int4.litertlm",
      "sha256": "9b0bd48b3695051274cd738c4a7931198e2f830fc268bca59573eab0a07d84a3",
      "size_mb": 5333.509
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "convert_nemotron_nano_9b.sh (hf-to-litertlm): nvidia/NVIDIA-Nemotron-Nano-9B-v2 at 6533e8de -> float export (6 prefill signatures 1024, 256, 64, 16, 4, 1 + decode, KV budget 4096, 62 state buffers = 4 KV pairs + 27 conv windows + 27 SSM states) -> int4 blockwise-32 (min-max) on the linear layers with every block scale floored at 2^-14 before quantizing, int8 lm_head and embedding -> metadata (the bundle template, no start token, stop ids 2 / 11 / 12, the thought channel, fp32 GPU activations, max_num_tokens 4096)",
    "quantization": "int4 blockwise-32 (min-max) on the linear layers, block scales floored at 2^-14 (the plain build left 66,388 zero or subnormal fp16 block scales of 240,988,160, which XNNPACK refuses), int8 lm_head and embedding; fp32 GPU activations declared (card L41, L118-L119)",
    "tool": "litert-torch (commit 115a136 + the Nemotron-H patch of hf-to-litertlm nemotron_h_work/nemotron_h_litert_torch.patch + a second patch for NVIDIA's dt arithmetic; lightweight export, 1e8-byte lazy-constant threshold); repro = hf-to-litertlm nemotron_h_work/convert_nemotron_nano_9b.sh",
    "tool_version": "litert-torch 115a136 + Nemotron-H patch + dt patch (torch 2.12.1, transformers 5.14.1, ai-edge-quantizer 0.8.0, litert-converter 0.3.0); packed and checked with litert-lm 0.18.0"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": 13.985,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes; the file = NVIDIA-Nemotron-Nano-9B-v2_int4.litertlm = the Hub file): litert-lm benchmark NVIDIA-Nemotron-Nano-9B-v2_int4.litertlm --backend cpu -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no",
            "pass 1: prefill 57.51 / decode 14.14 tok/s, TTFT 17.8749 s, init 17.8903 s",
            "pass 2: prefill 56.22 / decode 13.83 tok/s, TTFT 18.2944 s, init 17.637 s",
            "this file's benchmark process footprint (swap guard, 15 s samples, the largest of the two passes; footprint's binary units read as 2^30 B and written in 1e9 B): 33.3 GB, sample: 12:14:47 swap_used_MB 1824.50 unused 36G reclaimable_GB 82.97 footprint 52099:9024KB 52132:1792KB 52138:9137KB 79113:1888KB 8507:31GB; no guard kill"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "guard_footprint_max_gb": 33.3,
            "init_s": 17.7636,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 14.14,
            "pass1_init_s": 17.8903,
            "pass1_prefill_tok_s": 57.51,
            "pass1_ttft_s": 17.8749,
            "pass2_decode_tok_s": 13.83,
            "pass2_init_s": 17.637,
            "pass2_prefill_tok_s": 56.22,
            "pass2_ttft_s": 18.2944,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 36.3175
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 56.865,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 18084.6
        },
        "source": "data/device_runs/0.18.0/2026-10-15/nemotron-nano-9b-v2-int4__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": 45.52,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes; the file = NVIDIA-Nemotron-Nano-9B-v2_int4.litertlm = the Hub file): litert-lm benchmark NVIDIA-Nemotron-Nano-9B-v2_int4.litertlm --backend gpu -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no",
            "pass 1: prefill 474.95 / decode 44.44 tok/s, TTFT 2.1791 s, init 13.8777 s",
            "pass 2: prefill 481.58 / decode 46.6 tok/s, TTFT 2.1478 s, init 13.6284 s"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_s": 13.753,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 44.44,
            "pass1_init_s": 13.8777,
            "pass1_prefill_tok_s": 474.95,
            "pass1_ttft_s": 2.1791,
            "pass2_decode_tok_s": 46.6,
            "pass2_init_s": 13.6284,
            "pass2_prefill_tok_s": 481.58,
            "pass2_ttft_s": 2.1478,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 7.7682
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 478.265,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 2163.5
        },
        "source": "data/device_runs/0.18.0/2026-10-15/nemotron-nano-9b-v2-int4__mac-studio-m4-max.json"
      },
      {
        "device": "pixel-8a",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Pixel 8a",
            "machine_label": "pixel-8a-aab2",
            "os_build": "Android 16 (CP1A.260505.005)",
            "runtime": "litert-lm",
            "runtime_version": "prebuilt-adac974b",
            "soc": "Tensor G3",
            "vendor_sdk": null
          },
          "error": "GUARD_KILL avail=382932 pid=24383",
          "evidence": [
            "binary: Google's prebuilt litert_lm_advanced_main + 6 .so (gs://litert/binaries/latest/android_arm64/litert_lm/, object dated 2026-09-18, sha256 adac974b…)",
            "[16:34:52] bundle on device: 9b0bd48b3695051274cd738c4a7931198e2f830fc268bca59573eab0a07d84a3 (5592589856 B, push 120.4 s) = Mac",
            "one attempt by rule (--backend=gpu, text prompt 'What is the capital of France? Answer in one sentence. /no_think', --max_output_tokens=400); a 2-second sampler stops the process when MemAvailable falls under 400,000 kB",
            "VERBOSE: Replacing 3582 out of 3582 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (prefill_1024).",
            "W0000 00:00:1792049696.810634   24383 gpu_model_builder.cc:4455] Weights preparation on Gpu is disabled for PowerVR, Broadcom, Mali GPUs and non-OpenCL/WebGPU/Metal backends.",
            "[16:35:40] prompt_1 done: rc(adb)=0 EXIT=143 wall 42.7 s",
            "GUARD_KILL avail=382932 pid=24383",
            "parsed: i4_gpu_try gpu: gate FAIL 0/1 | answers: ''",
            "source: = transcript copies of what the run's files printed before its own dry run deleted the run dir at 18:04 the same day (README.md there = the source and what is lost: gpu_leg.json, samples 4-14 of 17, the full logs)",
            "OS build and SoC (the same phone 3 minutes earlier in the same boot: boot epoch 1792028616.3 there, 1792028616.1 here)"
          ],
          "failure_class": "memory_cap_exceeded",
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": false,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "memavail_at_start_kib": 4321480.0,
            "memavail_min_kib": 382932.0,
            "sampler_samples": 17.0,
            "seconds_to_stop": 42.7,
            "signatures_compiled": 1.0,
            "uptime_after_s": 21122.87,
            "uptime_before_s": 21076.0,
            "vmhwm_kib": 2608204.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": false,
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/prebuilt-adac974b/2026-10-15/nemotron-nano-9b-v2-int4__pixel-8a.json"
      }
    ]
  },
  "model": {
    "family": "nemotron-h",
    "id": "nemotron-nano-9b-v2-int4",
    "license": "other (nvidia-open-model-license)",
    "source_url": "https://huggingface.co/litert-community/NVIDIA-Nemotron-Nano-9B-v2",
    "task": "text-generation"
  },
  "pitfalls": [
    "Reasoning model: it opens a <think> block unless told otherwise; the bundle declares the thought channel (<think>\\n / \\n</think>), so the runtime returns the reasoning separately. NVIDIA's switch is text: /no_think in the system message or a user message turns the reasoning off (the template then ends the prompt with <think></think>), /think turns it back on; the CLI's --thinking false leaves the output byte for byte the same (the template reads no enable_thinking variable) (card L36, L88).",
    "Block scales: XNNPACK prepares a blockwise int4 layer only when every scale is a normal float; the plain int4 build left 66,388 zero or subnormal fp16 block scales, so this file floors them at 2^-14 before quantizing (card L118).",
    "GPU precision: the file declares fp32 activations; with the runtime's default fp16 activations the int8 weights answered none of the 8 check questions (every reply <unk> repeated to the cap) (card L56, L73, L119).",
    "Accuracy reference: GSM8K n=100 (thinking off, greedy, Mac GPU) int4 92 / int8 93 against transformers on MPS 83 (bf16) / 84 (fp32); the whole gap is on the 16 prompts longer than one 128-token Mamba2 chunk, where transformers' torch scan (no CUDA kernels) splits the prefill: re-run with the whole prompt in one chunk the reference scored the files' 15 of 16 (card L43, L64-L73).",
    "dt arithmetic: transformers 5.14.1's torch code path (what runs on a CPU or an Apple GPU) clamps each Mamba2 dt at 0.001 from below; its CUDA-kernel path passes the config's (0, ∞) limit instead, as NVIDIA's own modeling code does; the files follow NVIDIA's code (339 of 3,456 heads rest below 0.001) (card L116).",
    "Tool calling: the template renders NVIDIA's tool form, and the model writes <TOOLCALL>[{\"name\": ..., \"arguments\": ...}]</TOOLCALL> as text; the generic model type has no parser for it, so an app parses the calls itself (card L90).",
    "One Engine per conversation: litert-lm 0.18.0 starts a new Conversation on a used Engine from the earlier conversation's recurrent state (LiteRT-LM issue 3165); the 27 Mamba2 states make this file exposed to it (card L92).",
    "Conversation preface with tools and no messages: LiteRT-LM renders the chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with 'Failed to apply template: undefined value' when the preface is prefilled at creation (prefillPrefaceOnInit, default false; creating the conversation fails) and in the preface render (renderPrefaceIntoString and its C++ and C API forms); the same with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2); read in the source of LiteRT-LM 0.17.0, 0.18.1 and main, measured with the 0.18.1 PyPI package through its C API on these files' template text; the default path, where the first message is rendered with the preface, is not affected (the 0.18.1 Python API and CLI expose neither the flag nor the preface render); a system message or any initial message in the conversation config avoids it (card L123).",
    "Galaxy S26: not run — a sibling Mamba2-hybrid file of the same size class (Falcon-H1R-7B int4, 4.69 GB, the same binary) took the phone off USB 69 s into its GPU engine warm-up on 2026-10-15, and one such drop ends the tries on that phone; the CPU multipliers put the file above the phone's memory (card L109).",
    "Pixel 8a: the one GPU try was stopped by the harness's memory rule at MemAvailable 382,932 kB, 41 s after the start, after prefill_1024 was delegated 3582/3582 to the OpenCL delegate and while the runtime prepared the weights on the CPU side (Mali); no restart; GPU execution on a Mali phone is not verified (card L110).",
    "Shape: a three-signature variant (prefill 1024 / 256 / 1, the same weights) took the same time on the 1024 / 256 workload with a faster engine init (9.0 vs 13.8 s) and a slower first token on a 15-token prompt; the six-signature file ships (card L105).",
    "iOS not measured: the file is one 5.21 GiB TFLite section, over the ~2 GiB single-section limit (card L52).",
    "Mac: the int4 file is the faster file on the GPU (7.77 vs 8.37 s, decode 45.5 vs 41.0 tok/s) and the slower one on the CPU (its blockwise prefill 56.9 vs 127.9 tok/s); without --cache, the first CPU `run` writes a 4.94 GB compiled-weights cache next to the model (card L52, L86)."
  ],
  "schema_version": "1.2"
}
