{
  "artifacts": [
    {
      "file": "MNN-Pocket-0.3B_qualcomm_npu_ekv2048.litertlm",
      "sha256": "e8af457e3d2840a1c5b35f66e7e8bf17d1c88212608f2ec0c374ef01966060a0",
      "size_mb": 356.878
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "taobao-mnn/MNN-Pocket-0.3B rev 087a52fd -> npu_export, prefill signature 128, KV cache 2,048; calibration on 32 questions from GSM8K's training split (in the 4-shot form of the GSM8K check) and the first 16 of litert-torch's sample calibration prompts; two changes to npu_export's output: the prefill attention mask broadcast over the query heads (557,056 bytes instead of 3,899,392) and the 11 ranges of the two rotary-position graphs raised to 2,047.9375 (card L39, L193, L196)",
    "quantization": "int8 weights (one scale per output channel) with 16-bit integer activations (static ranges); KV cache of 2,048 tokens; one prefill signature of 128 tokens; no compiled NPU code (the phone compiles it at engine creation); model type qwen2p5; the int8 embedding table held twice with the same values, for the input embedding and the output layer (card L33, L39, L41, L187)",
    "tool": "litert-torch at commit 6d4c622, npu_export in three steps (the int8 weights of dynamic_wi8_afp32, activation ranges from 48 prompts, static quantization of the activations to 16-bit integers), packed with litert-lm-builder 0.17.0; repro = hf-to-litertlm REPRODUCE.md, entry \"2026-11-07 — the Qualcomm NPU file\" (card L196, L201)",
    "tool_version": "litert-torch 6d4c622, litert-lm-builder 0.17.0; on the phone LiteRT-LM 0.19.1 built from the v0.19.1 tag with Qualcomm's dispatch and compiler-plugin libraries and the QNN libraries of QAIRT 2.47.0.260601 (card L67, L194, L196)"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "npu",
          "context_length": null,
          "date": "2026-11-07",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocket-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": "QAIRT 2.47.0.260601114230 (QNN JIT libraries)"
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_qualcomm_npu_ekv2048.litertlm` at main 6ea52447: LFS sha256 e8af457e3d2840a1c5b35f66e7e8bf17d1c88212608f2ec0c374ef01966060a0, 374213768 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: the PyPI litert-lm-api 0.19.1 Android wheel's Python API on the phone (liblitert-lm.so sha256 3bc61c88a5998a41cc0eb35c5b133c14e5635790290839452d35cc936231d903), with the Qualcomm dispatch and compiler-plugin libraries of the LiteRT-LM main 292d4e52c build, byte-identical to the v0.19.1 tag's, QAIRT 2.47.0.260601114230 (QNN JIT libraries); HW cache update on; one engine per prompt set (a QNN compile per process)",
            "57 / 100 (the source model in bf16 58; paired both / R only / NPU only / neither 51 / 7 / 6 / 36, McNemar p 1.0; cap hits 23; first 40: 24); runs v9-npu-k1a, v9-npu-k1b, v9-npu-k1c, v9-npu-k1d, v9-npu-k1e in holds h1, h2 (l; every run's model_sha256 = the artifact); verdict PASS",
            "Every run ran with the main graph whole on the HTP (prefill_128 1,477 / 1,477 and decode 1,376 / 1,376 ops, each with its DispatchDelegate line); the logs put 2 of 13 nodes of prefill_rope_128, 2 of 12 of decode_rope and 1 of 4 of each embedder on XNNPACK, the rest of those graphs on the CPU kernels"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "gsm8k100_bar": 53.0,
            "gsm8k100_correct": 57.0,
            "gsm8k100_reference_bf16_correct": 58.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "GSM8K-100 on the NPU through the litert-lm-api 0.19.1 Android wheel (the test split's first 100, 4-shot chain of thought, greedy, cap 1,024 new tokens, one pass); bar 53 = the source model in bfloat16 (58) minus 5",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-qualcomm-npu-ekv2048__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "npu",
          "context_length": null,
          "date": "2026-11-07",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocket-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": "QAIRT 2.47.0.260601114230 (QNN JIT libraries)"
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_qualcomm_npu_ekv2048.litertlm` at main 6ea52447: LFS sha256 e8af457e3d2840a1c5b35f66e7e8bf17d1c88212608f2ec0c374ef01966060a0, 374213768 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "L (holds a1, a2) ITEMJSON full texts: items 50, equal 50 (q8 8 / 8, mt 2 / 2, gsm 40 / 40), different 0, missing 0",
            "every one of the 8 short questions stopped at the stop token (8 to 21 tokens out); in both conversations the third reply recalls the first turn"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "replies_compared": 50.0,
            "replies_equal": 50.0
          },
          "output_match": true,
          "peak_mem_mb": null,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the same replies through litert_lm_advanced_main built from the v0.19.1 tag (the speed record's binary and libraries): 8 short questions, two three-turn conversations and GSM8K questions 1-40, each full reply compared character for character with the Android wheel's reply",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-qualcomm-npu-ekv2048__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "npu",
          "context_length": 2048,
          "date": "2026-11-07",
          "decode_tokens_per_s": 147.2525,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-mnnpocket-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.1",
            "soc": "Qualcomm SM8850",
            "vendor_sdk": "QAIRT 2.47.0.260601114230 (QNN JIT libraries)"
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/MNN-Pocket-0.3B `MNN-Pocket-0.3B_qualcomm_npu_ekv2048.litertlm` at main 6ea52447: LFS sha256 e8af457e3d2840a1c5b35f66e7e8bf17d1c88212608f2ec0c374ef01966060a0, 374213768 B (anonymous Hub API tree read 2026-11-07 13:13:06 JST)",
            "runtime: LiteRT-LM v0.19.1 built from the tag (7a03724c, android_arm64, NDK 29.0.13113456): litert_lm_advanced_main sha256 5d869be730622b854fb87bf74205c7b083edece8197c9279317749dfd26772b7, the tag's Qualcomm dispatch libLiteRtDispatch_Qualcomm.so 6946f3771068302a894fbaaff53345f9460d211fe50b2866f2fadf388919f533 and compiler plugin libLiteRtCompilerPlugin_Qualcomm.so aab1b0e0be4ca6b78434562367de4a0fe43f02503be3d6aa1fb7b105fa2b98b3; QAIRT 2.47.0.260601114230 (QNN JIT libraries) libraries and skel; `--use_hw_cache_update_for_npu=true`; the binary's builds",
            "command: --backend=npu --input_prompt_file=<one-line file> --max_num_tokens=2048 --max_output_tokens=256 --async=false --use_session=false --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=3 (the cell = the mean of runs 2-3 in each of two processes, then the mean of the two; the binary prints TTFT to 0.01 s)",
            "Device: model=SM-S942Q release=16 build_id=BP4A.251205.006 fingerprint=samsung/m1qjpnx/m1q:16/BP4A.251205.006/S942QOPS1AZH9_SJP1AZH9:user/release-keys",
            "v9-t1 (run a1, 2026-11-07 07:39:54 JST at the run's start, uncapped (thermal 1, no CPU cap, kgsl_pwrlevel 0); rested 140s since 07:37:35): runs 2-3 prefill 10939.36 / 10831.47, decode 149.22 / 146.84 tok/s, TTFT 0.1 / 0.1 s, Init Total 29211.51 ms; VmHWM 3346496 kB",
            "v9-t2 (run a1, 2026-11-07 07:41:39 JST at the run's start, uncapped (thermal 1, no CPU cap, kgsl_pwrlevel 0); rested 68s since 07:40:31): runs 2-3 prefill 10810.26 / 10626.82, decode 147.51 / 145.44 tok/s, TTFT 0.1 / 0.1 s, Init Total 29274.15 ms; VmHWM 3346984 kB",
            "INFO: [compiler_plugin.cc:1103] Applying plugin with SDK version: v2.47.0.260601114230",
            "INFO: [dispatch_delegate.cc:129] DispatchDelegate::Initialize prefill_128",
            "VERBOSE: Replacing 1 out of 1 node(s) with delegate (DispatchDelegate) node, yielding 1 partitions for subgraph 0 (prefill_128).",
            "INFO: [litert_dispatch_device_context.cc:248] Creating new QNN context for bytecode 0xb4000072522f1f80 (size 243429376)",
            "INFO: [dispatch_delegate.cc:129] DispatchDelegate::Initialize decode",
            "VERBOSE: Replacing 1 out of 1 node(s) with delegate (DispatchDelegate) node, yielding 1 partitions for subgraph 1 (decode).",
            "INFO: [litert_dispatch_device_context.cc:248] Creating new QNN context for bytecode 0xb400007260b18f80 (size 309768192)",
            "VERBOSE: Replacing 2 out of 13 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 3 partitions for subgraph 0 (prefill_rope_128).",
            "VERBOSE: Replacing 2 out of 12 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 3 partitions for subgraph 1 (decode_rope).",
            "VERBOSE: Replacing 1 out of 4 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 2 partitions for subgraph 0 (decode_embedder).",
            "VERBOSE: Replacing 1 out of 4 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 2 partitions for subgraph 1 (prefill_embedder_128).",
            "the compiler plugin's refusals in the same log (the aux graphs, which stay on the CPU): 56 lines `LiteRT Op #<n> 'DynamicUpdateSlice' (code=151) is not supported in Qualcomm Compiler.` (each followed by `QNN error=3110`)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "decode_tokens": 256.0,
            "init_total_ms": 29242.83,
            "iterations": 3.0,
            "max_num_tokens": 2048.0,
            "prefill_tokens": 1024.0,
            "run1_decode_tok_s": 148.03,
            "run1_init_total_ms": 29211.51,
            "run1_prefill_tok_s": 10885.415,
            "run1_vmhwm_kb": 3346496.0,
            "run2_decode_tok_s": 146.475,
            "run2_init_total_ms": 29274.15,
            "run2_prefill_tok_s": 10718.54,
            "run2_vmhwm_kb": 3346984.0,
            "runs": 2.0
          },
          "output_match": null,
          "peak_mem_mb": 3427.3116,
          "prefill_tokens_per_s": 10801.9775,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 100.0
        },
        "source": "data/device_runs/0.19.1/2026-11-07/mnn-pocket-0.3b-qualcomm-npu-ekv2048__galaxy-s26.json"
      }
    ]
  },
  "model": {
    "family": "mnn-pocket",
    "id": "mnn-pocket-0.3b-qualcomm-npu-ekv2048",
    "license": "apache-2.0",
    "source_url": "https://huggingface.co/litert-community/MNN-Pocket-0.3B",
    "task": "text-generation"
  },
  "pitfalls": [
    "Compile at every engine creation: each new engine compiles the file for the NPU before the first reply, 27 to 29 s on a Galaxy S26 whose CPU was not capped when the engine started, longer once earlier compiles have capped it; LiteRT-LM 0.19.1 keeps no compiled copy between processes (card L102). The process's peak memory reached 3.43 GB and the phone's MemAvailable fell to 4.12 GB during the compile (card L140, L143).",
    "Needs Android binaries built from source: LiteRT-LM 0.19.1 releases no Android build of the tool or of its NPU libraries, so build them from the v0.19.1 tag with Qualcomm's compiler plugin added and take the QNN libraries from QAIRT 2.47.0.260601 (card L67, L187).",
    "LiteRT-LM 0.19.1's CPU backend does not open the file, and no Mac backend computes its 16-bit integer graph the way the phone's NPU does, so there is no Mac reply to match word for word (card L39, L177).",
    "Greedy on the NPU: LiteRT-LM 0.19.1 picks the most likely token at every step on the NPU, so the file's sampling settings do not apply there (card L102).",
    "One prefill signature of 128 tokens: a longer prompt is prefilled 128 tokens at a time (card L188).",
    "Attention mask size: on the SM8850 NPU with QAIRT 2.47 an int16 mask concatenated to more than 1 MiB was added to the wrong rows (litert-torch#1184; QAIRT 2.50 adds the issue's two-operation case correctly), so the mask is broadcast over the query heads and stays under 1 MiB (card L196).",
    "Rotary-position ranges: the calibrated ranges ended at the largest positions the calibration reached (511 in prefill, 527 in decode); with them the cosines and sines were off by up to 1.99 past those positions and 31 of the first 40 GSM8K replies on the Galaxy S26 NPU ran to the 1,024-token limit, against 10 with the ranges raised to 2,047.9375 (card L196).",
    "CPU work around the NPU: the main graph runs whole on the NPU (1,477 operations of prefill, 1,376 of decode); the embedding lookup and the small graphs for the rotary position, the attention mask and the cache update run on the CPU, and the phone capped the CPU clocks during the compile and lowered them further during the runs (card L136, L145).",
    "One phone: the file was run on a Galaxy S26 (Snapdragon SM8850) only; another Snapdragon phone compiles it for its own NPU at engine start, untested (card L187)."
  ],
  "schema_version": "1.2"
}
