{
  "artifacts": [
    {
      "file": "Olmo-3-7B-Think_int8.litertlm",
      "sha256": "e075c5c92582fa1446723003225239b4dd131a07195ca49e4a7117388641bfbb",
      "size_mb": 7090.256
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "convert_olmo3_7b_think.sh (hf-to-litertlm): allenai/Olmo-3-7B-Think at d97e442d -> export (6 prefill signatures 1024..1 + decode, KV cache 4096, embedder externalized, the bundle template embedded) -> metadata edit (no start token, stop ids 100265 / 100257, the thought channel, max_num_tokens 4096) -> the repo's tokenizer.json as the tokenizer section",
    "quantization": "int8 dynamic on every linear layer (219 FULLY_CONNECTED per prefill signature, 225 in decode), int8 embedding table; activations float, the runtime's default fp16 on the GPU (card L31, L108)",
    "tool": "litert-torch (stock export_hf path, lightweight conversion, embedding table as its own section); repro = hf-to-litertlm olmo3_work/",
    "tool_version": "litert-torch 0.9.4 (litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1, torch 2.13.0); packed and checked with litert-lm 0.18.0"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-10-16",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.17.1",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (tools/v0171_run.sh, \"$B\" = Olmo-3-7B-Think_int8.litertlm): uvx --from litert-lm==0.17.1 litert-lm run \"$B\" --backend cpu --prompt \"What is the capital of France? Answer in one word.\" --cache no",
            "stdout: [thought] ... [/thought] then Paris",
            "EXIT=0",
            "process footprint (swap guard, 15 s samples, binary GB): max 46 GB; no guard kill",
            "2026-10-16 12:17:44 JST batch: 0.17.1 runs exit 0 (int8: EXIT=0 int4: EXIT=0 )"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "answers_with_expect": 1.0,
            "guard_footprint_max_gb": 46.0,
            "processes": 1.0
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.17.1/2026-10-16/olmo-3-7b-think-int8__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "cpu",
          "context_length": null,
          "date": "2026-10-16",
          "decode_tokens_per_s": 6.695,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes): litert-lm benchmark Olmo-3-7B-Think_int8.litertlm --backend cpu -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no",
            "pass 1: prefill 128.34 / decode 6.66 tok/s, TTFT 8.1481 s, init 50.4204 s",
            "pass 2: prefill 134.38 / decode 6.73 tok/s, TTFT 7.7695 s, init 50.8257 s",
            "process footprint (swap guard, 15 s samples, binary GB); no guard kill"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "guard_footprint_max_gb": 47.0,
            "init_s": 50.623,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 6.66,
            "pass1_init_s": 50.4204,
            "pass1_prefill_tok_s": 128.34,
            "pass1_ttft_s": 8.1481,
            "pass2_decode_tok_s": 6.73,
            "pass2_init_s": 50.8257,
            "pass2_prefill_tok_s": 134.38,
            "pass2_ttft_s": 7.7695,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 46.038
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 131.36,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 7958.8
        },
        "source": "data/device_runs/0.18.0/2026-10-16/olmo-3-7b-think-int8__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": null,
          "date": "2026-10-16",
          "decode_tokens_per_s": 30.395,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.18.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "command (both passes; one pass for the four files, the header): Olmo-3-7B-Think_int8.litertlm --backend gpu, prefill 1024, decode 256, max tokens 2048, --runs 3, --cache no",
            "pass 1: prefill 840.98 / decode 30.26 tok/s, TTFT 1.2507 s, init 4.5791 s",
            "pass 2: prefill 836.63 / decode 30.53 tok/s, TTFT 1.2567 s, init 4.5713 s",
            "each GPU pass logged 'TopK requires src tensor C dimension to be divisible by 4.' (4 / 4 lines: warmup + 3 iterations; vocabulary 100,278 = 2 mod 4)",
            "delegation from the verbose run of the same bytes (2026-10-16, the run r8 gate): Replacing 1436 out of 1436 node(s) with delegate (LITERT_WEBGPU) node, yielding 1 partitions for subgraph 0 (prefill_1024). .. Replacing 1336 out of 1336 node(s) with delegate (LITERT_WEBGPU) node, yielding 1 partitions for subgraph 6 (decode). (7 decoder signatures, 9823 / 9823 nodes)"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_s": 4.5752,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "pass1_decode_tok_s": 30.26,
            "pass1_init_s": 4.5791,
            "pass1_prefill_tok_s": 840.98,
            "pass1_ttft_s": 1.2507,
            "pass2_decode_tok_s": 30.53,
            "pass2_init_s": 4.5713,
            "pass2_prefill_tok_s": 836.63,
            "pass2_ttft_s": 1.2567,
            "passes": 2.0,
            "prefill_tokens": 1024.0,
            "t_1024_256_s": 9.6434
          },
          "output_match": null,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": 838.805,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 1253.7
        },
        "source": "data/device_runs/0.18.0/2026-10-16/olmo-3-7b-think-int8__mac-studio-m4-max.json"
      }
    ]
  },
  "model": {
    "family": "olmo3",
    "id": "olmo-3-7b-think-int8",
    "license": "apache-2.0",
    "source_url": "https://huggingface.co/litert-community/Olmo-3-7B-Think",
    "task": "text-generation"
  },
  "pitfalls": [
    "Always thinks: the template ends the prompt with an opened <think>; the CLI's --thinking false and --thinking-budget 0 leave the output byte for byte the same, and a positive budget closes the thought channel at the budget while the model goes on reasoning in the visible text (with --thinking-budget 64 it wrote </think> as text, then the answer) (card L27, L79).",
    "Long reasoning inside a 4,096-token context: prompt, thought and answer share the KV budget; on GSM8K the mean reply was about 1,600 tokens and the 0.18.0 CLI has no output-length flag, so a long thought can reach the end of the context before the answer (card L27, L79).",
    "Accuracy: GSM8K n=100 (greedy, 2,048 output tokens, answers scored after </think>, a never-closed reply scored on its whole text, Mac GPU): bf16 84 / int8 81, unclosed thoughts 26 / 32 (card L57-L66).",
    "Chat template: Ai2's chat_template.jinja with three edits for the runtime's template engine (a content-parts macro, every content read through it, message.get(...) written as `is defined` tests); Ai2's default system message from the template file (\"You are OLMo, a helpful function-calling AI assistant ...\"), which differs from the README's \"Default System Message\" section; <think> / </think> are three plain tokens each (card L81).",
    "No start token; stop ids 100265 (<|im_end|>) and 100257 (<|endoftext|>); the bundle sets no sampler (every card number greedy; Ai2 recommends temperature 0.6 / top_p 0.95) (card L79, L81, L109).",
    "Tool calling is not wired: the runtime's tools= argument is not rendered (the template has no tools branch) and the generic model type has no tool-call parser; with Ai2's `functions` form this file (CPU) answered in text; a result in an `environment` turn got no answer inside 2,048 tokens (card L83).",
    "GPU sampler: every GPU run logs `TopK requires src tensor C dimension to be divisible by 4.` (vocabulary 100,278); the last two ids (<|endofprompt|>, <|pad|>) can never be sampled on the GPU (card L51).",
    "Precision: the same weights with fp32 GPU activations declared took 41 % more time (13.60 vs 9.64 s) and gave the same eight check answers, so the file keeps the runtime's fp16 default (card L96).",
    "Graph: the exporter's graph (no GPU composites): the GPU-composites rewrite that sped up two dense 8B files of the same converter refuses this graph (the attention-mask input also feeds the sliding-window mask); every layer keeps a full-length 4,096 cache, which equals the sliding window (card L25, L107).",
    "Galaxy S26: the int8 file was not started on the phone's GPU (a sibling 8B-class int8 file restarted the phone during its GPU engine warm-up) and the CPU was not run (peak arithmetic above the phone's memory); Pixel 8a not run; not run on an NPU (card L100-L102).",
    "iOS not measured: the decoder graph is one 6.53 GiB TFLite section, over the ~2 GiB single-section limit (card L45).",
    "Mac CPU: without --cache, `run` writes a compiled-weights cache next to the model on its first CPU run (6.90 GB for this file); the int8 file is the faster file on the Mac CPU (46.04 vs 51.97 s), the int4 file on the GPU (card L45, L79)."
  ],
  "schema_version": "1.2"
}
