{
  "artifacts": [
    {
      "file": "NVIDIA-Nemotron-Nano-9B-v2_int8.litertlm",
      "sha256": "575505ddb315537b347ad1b5bc3385f5770f4c0152840884ae825e394ed3a194",
      "size_mb": 8651.981
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "convert_nemotron_nano_9b.sh (hf-to-litertlm): nvidia/NVIDIA-Nemotron-Nano-9B-v2 at 6533e8de -> float export (6 prefill signatures 1024, 256, 64, 16, 4, 1 + decode, KV budget 4096, 62 state buffers = 4 KV pairs + 27 conv windows + 27 SSM states) -> dynamic int8 on every FULLY_CONNECTED + int8 embedding -> metadata (the bundle template, no start token, stop ids 2 / 11 / 12, the thought channel, fp32 GPU activations, max_num_tokens 4096)",
    "quantization": "int8 dynamic on every linear layer, int8 embedding; the Mamba conv and the selective scan float; fp32 GPU activations declared (card L40, L118-L119)",
    "tool": "litert-torch (commit 115a136 + the Nemotron-H patch of hf-to-litertlm nemotron_h_work/nemotron_h_litert_torch.patch + a second patch for NVIDIA's dt arithmetic; lightweight export, 1e8-byte lazy-constant threshold); repro = hf-to-litertlm nemotron_h_work/convert_nemotron_nano_9b.sh",
    "tool_version": "litert-torch 115a136 + Nemotron-H patch + dt patch (torch 2.12.1, transformers 5.14.1, ai-edge-quantizer 0.8.0, litert-converter 0.3.0); packed and checked with litert-lm 0.18.0"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": 12.39,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes; the file = NVIDIA-Nemotron-Nano-9B-v2_int8.litertlm = the Hub file): litert-lm benchmark NVIDIA-Nemotron-Nano-9B-v2_int8.litertlm --backend cpu -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no",
            "pass 1: prefill 128.54 / decode 12.67 tok/s, TTFT 8.0459 s, init 60.0238 s",
            "pass 2: prefill 127.32 / decode 12.11 tok/s, TTFT 8.1256 s, init 58.6313 s",
            "this file's benchmark process footprint (swap guard, 15 s samples, the largest of the two passes; footprint's binary units read as 2^30 B and written in 1e9 B): 56.9 GB, sample: 12:11:21 swap_used_MB 1824.50 unused 12G reclaimable_GB 71.82 footprint 52099:9024KB 52132:1792KB 52138:9137KB 79113:1872KB 79121:53GB; no guard kill"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "guard_footprint_max_gb": 56.9,
            "init_s": 59.3276,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 12.67,
            "pass1_init_s": 60.0238,
            "pass1_prefill_tok_s": 128.54,
            "pass1_ttft_s": 8.0459,
            "pass2_decode_tok_s": 12.11,
            "pass2_init_s": 58.6313,
            "pass2_prefill_tok_s": 127.32,
            "pass2_ttft_s": 8.1256,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 28.6769
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 127.93,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 8085.8
        },
        "source": "data/device_runs/0.18.0/2026-10-15/nemotron-nano-9b-v2-int8__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-14",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.17.1",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "job: v0171 gpu What is the capital of France? Answer briefly. /no_think",
            "command (tools/run_job.sh, kind v0171): litert-lm run <cand int8> --backend gpu --prompt <the prompt> --cache no --top-k 1 --seed 0 --max-num-tokens 4096 (no --verbose)",
            "stdout tail: The capital of France is Paris.",
            "exit 0, Validation error 0, Shape mismatch 0"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "answers_with_expect": 1.0,
            "processes": 1.0,
            "seconds": 19.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.17.1/2026-10-14/nemotron-nano-9b-v2-int8__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-15",
          "decode_tokens_per_s": 41.02,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes; the file = NVIDIA-Nemotron-Nano-9B-v2_int8.litertlm = the Hub file): litert-lm benchmark NVIDIA-Nemotron-Nano-9B-v2_int8.litertlm --backend gpu -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no",
            "pass 1: prefill 480.46 / decode 41.49 tok/s, TTFT 2.1554 s, init 12.2013 s",
            "pass 2: prefill 479.89 / decode 40.55 tok/s, TTFT 2.1585 s, init 12.2104 s"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_s": 12.2058,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 41.49,
            "pass1_init_s": 12.2013,
            "pass1_prefill_tok_s": 480.46,
            "pass1_ttft_s": 2.1554,
            "pass2_decode_tok_s": 40.55,
            "pass2_init_s": 12.2104,
            "pass2_prefill_tok_s": 479.89,
            "pass2_ttft_s": 2.1585,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 8.3742
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 480.175,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 2157.0
        },
        "source": "data/device_runs/0.18.0/2026-10-15/nemotron-nano-9b-v2-int8__mac-studio-m4-max.json"
      }
    ]
  },
  "model": {
    "family": "nemotron-h",
    "id": "nemotron-nano-9b-v2-int8",
    "license": "other (nvidia-open-model-license)",
    "source_url": "https://huggingface.co/litert-community/NVIDIA-Nemotron-Nano-9B-v2",
    "task": "text-generation"
  },
  "pitfalls": [
    "Reasoning model: it opens a <think> block unless told otherwise; the bundle declares the thought channel (<think>\\n / \\n</think>), so the runtime returns the reasoning separately. NVIDIA's switch is text: /no_think in the system message or a user message turns the reasoning off (the template then ends the prompt with <think></think>), /think turns it back on; the CLI's --thinking false leaves the output byte for byte the same (the template reads no enable_thinking variable) (card L36, L88).",
    "GPU precision: the file declares fp32 activations; with the runtime's default fp16 activations it answered none of the 8 check questions (every reply <unk> repeated to the cap), for 1.5 % less time (card L56, L73, L119).",
    "Accuracy reference: GSM8K n=100 (thinking off, greedy, Mac GPU) int8 93 / int4 92 against transformers on MPS 83 (bf16) / 84 (fp32); the whole gap is on the 16 prompts longer than one 128-token Mamba2 chunk, where transformers' torch scan (no CUDA kernels) splits the prefill: re-run with the whole prompt in one chunk the reference scored the files' 15 of 16 (card L43, L64-L73).",
    "dt arithmetic: transformers 5.14.1's torch code path (what runs on a CPU or an Apple GPU) clamps each Mamba2 dt at 0.001 from below; its CUDA-kernel path passes the config's (0, ∞) limit instead, as NVIDIA's own modeling code does; the files follow NVIDIA's code (339 of 3,456 heads rest below 0.001); an int8 build with the transformers clamp scored the same 93 on GSM8K and differs by one MAXIMUM op per mixer (card L116).",
    "Tool calling: the template renders NVIDIA's tool form, and the file writes <TOOLCALL>[{\"name\": ..., \"arguments\": ...}]</TOOLCALL> as text; the generic model type has no parser for it, so an app parses the calls itself (card L90).",
    "One Engine per conversation: litert-lm 0.18.0 starts a new Conversation on a used Engine from the earlier conversation's recurrent state (LiteRT-LM issue 3165); the probe on this file gave the same answers on a shared and a fresh Engine, but the 27 Mamba2 states are there all the same (card L92).",
    "Chat template: NVIDIA's template does not parse on the 0.18.0 runtime (its rewrite of .split('</think>')[1] is a syntax error); the bundle's template is NVIDIA's with eleven edits (content parts, the string methods as filters, the trailing-assistant special case removed), equal to transformers' render in 39 of 39 cells; no start token; stop ids 2 / 11 / 12; the tokenizer section is the repo's tokenizer.json byte for byte (card L88).",
    "Conversation preface with tools and no messages: LiteRT-LM renders the chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with 'Failed to apply template: undefined value' when the preface is prefilled at creation (prefillPrefaceOnInit, default false; creating the conversation fails) and in the preface render (renderPrefaceIntoString and its C++ and C API forms); the same with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2); read in the source of LiteRT-LM 0.17.0, 0.18.1 and main, measured with the 0.18.1 PyPI package through its C API on these files' template text; the default path, where the first message is rendered with the preface, is not affected (the 0.18.1 Python API and CLI expose neither the flag nor the preface render); a system message or any initial message in the conversation config avoids it (card L123).",
    "Galaxy S26: not run — a sibling Mamba2-hybrid file of the same size class (Falcon-H1R-7B int4, 4.69 GB) took the phone off USB 69 s into its GPU engine warm-up on 2026-10-15, and no 8B-class int8 engine is started on that phone after an 8B int8 file restarted it during its GPU warm-up; the CPU multipliers put both files above the phone's memory (card L109).",
    "Pixel 8a: the int8 file was not tried (9.07 GB, larger than the phone's RAM); NPU not run (no HTP path for dynamic-range int8) (card L110-L111).",
    "iOS not measured: the file is one 8.45 GiB TFLite section, over the ~2 GiB single-section limit (card L52).",
    "Mac CPU: without --cache, `run` writes a compiled-weights cache next to the model on its first CPU run (8.32 GB for this file); the int8 file is the faster file on the Mac CPU (28.68 vs 36.32 s), the int4 file on the GPU (card L52, L86)."
  ],
  "schema_version": "1.2"
}
