{
  "artifacts": [
    {
      "file": "Mage-VL-Cowork_vision-fp16.litertlm",
      "sha256": "1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df",
      "size_mb": 3051.369
    }
  ],
  "benchmarks": [],
  "conversion": {
    "command": "microsoft/Mage-VL-Cowork rev 6fbe7f91 -> PY_PREP=... PY_EXPORT=... PY_SAMPLE=... LITERT_LM=... bash scripts/reproduce_vlm.sh mage-vl-cowork of hf-to-litertlm (the key that also builds the sibling file; run as written on 2026-11-04, both files section by section the same as the shipped bytes)",
    "quantization": "the sibling's decoder, embedding table and adapter (text decoder int4, blockwise, 128 per block, OCTAV, a 4,096-token KV cache, fp16 on the GPU through an exact rotary-position path; int8 embedding table); a vision encoder with fp16 fully connected weights and biases and the model metadata reduced_precision_support = fp16accfp16 for XNNPACK's fp16 path on the CPU (card L31, L309-L311, L325)",
    "tool": "litert-torch 0.9.3 (the checkpoint's Qwen3-4B text model as a standalone Qwen3ForCausalLM with the GPU's cache-update and attention composites; the vision encoder and 2 x 2 merger for one static picture, its LayerNorm inputs scaled and its fully connected weights stored as fp16) and litert-lm-builder 0.16.1 as a fast_vlm bundle; repro = hf-to-litertlm scripts/ship_mage_vl_cowork.sh (card L309-L311, L325-L328)",
    "tool_version": "litert-torch 0.9.3, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, ai-edge-litert 2.2.0, litert-lm-builder 0.16.1, transformers 5.14.1, torch 2.13.0, torchao 0.18.0 on Python 3.14.6 for the export; torch 2.12.1, transformers 5.14.1, tokenizers 0.22.2, safetensors 0.8.0 on Python 3.12.13 for the preparation; litert-lm 0.19.0 for the runs (recipe entry)"
  },
  "cross_runtime": [],
  "delegation": null,
  "device": {
    "records": [
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "cpu",
          "context_length": 4096,
          "date": "2026-11-04",
          "decode_tokens_per_s": 5.16,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-magevl-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/Mage-VL-Cowork `Mage-VL-Cowork_vision-fp16.litertlm` at main 38bf428b: LFS sha256 1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df, 3199591888 B (anonymous Hub API tree read 2026-11-05 03:21 JST)",
            "runtime: PyPI litert-lm-api 0.19.0 Android wheel (liblitert-lm.so sha256 28fa8433febc49a99f4bd3ef8cb82c2912457cfe274d97944debeabdeb97fc78) on CPython 3.14.8, the Python API on the phone: `Engine(model, backend=CPU(thread_count=4), vision_backend=CPU(), max_num_images=1, enable_benchmark=True); create_conversation(messages=[system]); send_message_async({'role': 'user', 'content': [image, text]}, max_output_tokens=1108)`; sampling: the bundle declares no sampler, so the runtime's CPU / GPU default applies (top-k 1 = greedy)",
            "protocol: a benchmark script, one process per run, the phone's state and a ready check before each run, the process stopped by the memory guard on MemAvailable",
            "the one request, the decoder and the vision encoder on the CPU (XNNPACK, 4 threads): ScreenSpot-v2 desktop_006 stretched to 1920 x 1088 + the vendor's GUI agent prompt = 2,988 tokens in, a fresh process each (a new Engine); runs: Vk4e-cpu-w2 (svea, 2026-11-04 21:09 JST, cache keep, thermal 3 before; check run, not waited for clean (thermal 3, cap policy0:1228800/3628800 policy6:1267200/4742400 , kgsl_pwrlevel 7)); Vk4e-cpu-w1 (svea, 2026-11-04 21:06 JST, cache cold, thermal 3 before; check run, not waited for clean (thermal 3, cap policy0:2112000/3628800 policy6:2112000/4742400 , kgsl_pwrlevel 6)); every run's model_sha256 = the artifact; the cell = the mean over the warm processes (each in metrics: request_s = from before Engine() to the end of the reply; engine_only_s = Engine(); conversation_s = create_conversation())",
            "ttft_ms = the runtime's time to first token, which does not hold the vision stage (first chunk less it, in metrics)",
            "full_delegation false = XNNPACK takes a part of each decoder signature; the log gives two lines per signature (which set ran is not read), the composites' subgraphs apart",
            "both processes ran right after a GPU run with no wait: thermal status 3 at the start, under different CPU caps (2.11 GHz cold, 1.23-1.27 GHz warm), so they do not compare the caches",
            "delegation (Vk4e-cpu-w2): prefill_1024 1468 / 1729 on TfLiteXNNPackDelegate (355 partition(s)); decode 1389 / 1654 on TfLiteXNNPackDelegate (362 partition(s)); main 1 / 4 on TfLiteXNNPackDelegate (2 partition(s)); decode 2439 / 2523 on TfLiteXNNPackDelegate (74 partition(s)); main 7915 / 7939 on TfLiteXNNPackDelegate (49 partition(s)); main 20 / 26 on TfLiteXNNPackDelegate (3 partition(s)); prefill_1024 2568 / 2653 on TfLiteXNNPackDelegate (73 partition(s))",
            "peak_mem_mb = the highest VmHWM of these processes in MiB (phone-side samples, kB / 1024); it does not hold the GPU's memory",
            "the card: https://huggingface.co/litert-community/Mage-VL-Cowork (Performance)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "conversation_s_cold": 1.444,
            "conversation_s_p1": 0.364,
            "decode_tokens_cold": 297.0,
            "decode_tokens_p1": 297.0,
            "decode_tokens_per_s_cold": 5.33,
            "decode_tokens_per_s_p1": 5.16,
            "engine_only_s_cold": 3.886,
            "engine_only_s_p1": 2.919,
            "first_chunk_s_cold": 99.184,
            "first_chunk_s_p1": 122.278,
            "memavailable_min_kb": 4452592.0,
            "memavailable_min_kb_cold": 4452592.0,
            "memavailable_min_kb_p1": 4745488.0,
            "prefill_tokens": 2988.0,
            "prefill_tokens_per_s_cold": 41.3,
            "prefill_tokens_per_s_p1": 34.1,
            "processes_cold": 1.0,
            "processes_warm": 1.0,
            "request_s_cold": 159.73,
            "request_s_p1": 182.69,
            "request_s_warm_mean": 182.69,
            "skin_first_c_cold": 45.0,
            "skin_first_c_p1": 44.8,
            "thermal_first_cold": 3.0,
            "thermal_first_p1": 3.0,
            "ttft_s_cold": 72.594,
            "ttft_s_p1": 87.803,
            "vision_stage_s_cold": 26.59,
            "vision_stage_s_p1": 34.48,
            "vmhwm_kb_cold": 5515752.0,
            "vmhwm_kb_p1": 5650804.0
          },
          "output_match": null,
          "peak_mem_mb": 5518.4,
          "prefill_tokens_per_s": 34.1,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 87802.8
        },
        "source": "data/device_runs/0.19.0/2026-11-04/mage-vl-cowork-vision-fp16__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": 4096,
          "date": "2026-11-04",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-magevl-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/Mage-VL-Cowork `Mage-VL-Cowork_vision-fp16.litertlm` at main 38bf428b: LFS sha256 1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df, 3199591888 B (anonymous Hub API tree read 2026-11-05 03:21 JST)",
            "runtime: PyPI litert-lm-api 0.19.0 Android wheel (liblitert-lm.so sha256 28fa8433febc49a99f4bd3ef8cb82c2912457cfe274d97944debeabdeb97fc78) on CPython 3.14.8, the Python API on the phone: `Engine(model, backend=GPU(), vision_backend=CPU(), max_num_images=1, enable_benchmark=True); create_conversation(messages=[system]); send_message_async({'role': 'user', 'content': [image, text]}, max_output_tokens=1108)`; sampling: the bundle declares no sampler, so the runtime's CPU / GPU default applies (top-k 1 = greedy)",
            "protocol: a benchmark script, one process per run, the phone's state and a ready check before each run, the process stopped by the memory guard on MemAvailable",
            "the test: 120 ScreenSpot-v2 screenshots (40 desktop, 40 web, 40 mobile, a fixed shuffle), each stretched to 1920 x 1088 and sent as the vendor's GUI agent's first step (MAGE_GUI_SYSTEM_PROMPT, `Instruction: Click on the element for: <the instruction>`, `Previous actions: None`), one fresh process per screenshot",
            "runs (120: 2026-11-04 17:16-2026-11-04 20:34 JST, runs svka, svkb, svkc, svkd, svke, svkf); every row's bundle_sha256 = the artifact; no wait for a cool phone inside a hold, so the seconds in metrics are the phone's sustained state, not a cold start: 108 of the 120 started at thermal status 3 (request = from before Engine() to the end of the reply)",
            "output_match = the line: a reply is right when its first tool call is a click whose point, converted to the screenshot's pixels, falls inside the target's box; per group the rate must reach the original model's rate in fp32 or bf16, whichever is lower, minus 5 points (the original in PyTorch on the Mac's GPU, MPS, transformers 4.57.3, greedy); and every reply's tool call parses as JSON, ends at <|im_end|> and has an Action: line after </think>; scores: desktop 62.5 % (bar 47.5), web 65.0 % (bar 57.5), mobile 60.0 % (bar 50.0), all 120 62.5 % against the original's 60.0 % in fp32 and 60.8 % in bf16; JSON 100.0 %, stop failures 0, form failures 0",
            "against the original on all 120 (right only in the original / right only in this form, exact McNemar p): fp32 9 / 12 (p 0.66), bf16 6 / 8 (p 0.79)",
            "full_delegation false = the decoder's 2 signatures whole on the GPU (LITERT_CL: prefill_1024 1,729 of 1,729 nodes, decode 1,654 of 1,654, 1 partition each); the vision encoder on the CPU as requested (XNNPACK 7,915 of 7,939 nodes, 49 partitions), the vision adapter (XNNPACK 20 of 26) and the embedder (XNNPACK 1 of 4) on the CPU, as the runtime log lists them",
            "kgsl_page_alloc_max_mb = the highest /sys/class/kgsl/kgsl/page_alloc (the phone's GPU pages, every process), in MiB; the card gives 4.49 GB",
            "delegation (Vk4-gv-desktop_006): prefill_1024 1729 / 1729 on LITERT_CL (1 partition(s)); decode 1654 / 1654 on LITERT_CL (1 partition(s)); main 1 / 4 on TfLiteXNNPackDelegate (2 partition(s)); main 7915 / 7939 on TfLiteXNNPackDelegate (49 partition(s)); main 20 / 26 on TfLiteXNNPackDelegate (3 partition(s))",
            "peak_mem_mb = the highest VmHWM of these processes in MiB (phone-side samples, kB / 1024); it does not hold the GPU's memory"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "all_pct": 62.5,
            "all_right": 75.0,
            "decode_tokens_per_s_median": 10.81,
            "desktop_bar_pct": 47.5,
            "desktop_original_bf16_pct": 65.0,
            "desktop_original_fp32_pct": 52.5,
            "desktop_pct": 62.5,
            "desktop_right": 25.0,
            "form_failures": 0.0,
            "items": 120.0,
            "kgsl_page_alloc_max_mb": 4283.9,
            "largest_position": 3411.0,
            "memavailable_min_kb": 2309364.0,
            "mobile_bar_pct": 50.0,
            "mobile_original_bf16_pct": 55.0,
            "mobile_original_fp32_pct": 62.5,
            "mobile_pct": 60.0,
            "mobile_right": 24.0,
            "n_gen_max": 419.0,
            "n_gen_p50": 259.0,
            "n_input_max": 2996.0,
            "n_input_min": 2986.0,
            "original_bf16_pct": 60.8,
            "original_fp32_pct": 60.0,
            "prefill_tokens_per_s_median": 146.6,
            "request_s_max": 101.6,
            "request_s_mean": 82.98,
            "request_s_median": 83.8,
            "request_s_min": 56.6,
            "stop_failures": 0.0,
            "tool_call_json_pct": 100.0,
            "web_bar_pct": 57.5,
            "web_original_bf16_pct": 62.5,
            "web_original_fp32_pct": 65.0,
            "web_pct": 65.0,
            "web_right": 26.0
          },
          "output_match": true,
          "peak_mem_mb": 2715.3,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the GUI agent line (decoder on the GPU, vision encoder on the CPU): 120 ScreenSpot-v2 screenshots, one fresh process each, each click judged against the target's box; bar per group = the original model's lower rate (fp32 or bf16) minus 5 points",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.0/2026-11-04/mage-vl-cowork-vision-fp16__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": 4096,
          "date": "2026-11-04",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-magevl-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/Mage-VL-Cowork `Mage-VL-Cowork_vision-fp16.litertlm` at main 38bf428b: LFS sha256 1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df, 3199591888 B (anonymous Hub API tree read 2026-11-05 03:21 JST)",
            "runtime: PyPI litert-lm-api 0.19.0 Android wheel (liblitert-lm.so sha256 28fa8433febc49a99f4bd3ef8cb82c2912457cfe274d97944debeabdeb97fc78) on CPython 3.14.8, the Python API on the phone: `Engine(model, backend=GPU(), vision_backend=CPU(), max_num_images=1, enable_benchmark=True); create_conversation(messages=[system]); send_message_async({'role': 'user', 'content': [image, text]}, max_output_tokens=910)`; sampling: the bundle declares no sampler, so the runtime's CPU / GPU default applies (top-k 1 = greedy)",
            "protocol: a benchmark script, one process per run, the phone's state and a ready check before each run, the process stopped by the memory guard on MemAvailable",
            "the test: the longest requests the work has, one process each: the vendor's hybrid system prompt as written with the picture and text of desktop_006 (3,186 tokens), and three requests with 40 earlier-step lines (desktop_006 and web_013 3,546 tokens, mobile_491 3,547), which pass fp16's exact-integer limit of 2,048 on the decoder's rotary path; output_match stays null: the clause asks that the whole prompt prefills, the reply ends at the stop token and its tool call parses, not that it matches a reference",
            "runs: Vk4e-gv-e_hyb_desktop_006 (svea, 2026-11-04 20:59 JST, cache cold, thermal 0 before; clean); Vk4e-gv-e_long_desktop_006 (svea, 2026-11-04 21:00 JST, cache keep, thermal 1 before; check run, not waited for clean (thermal 1, cap policy6:4396800/4742400 , kgsl_pwrlevel 0)); Vk4e-gv-e_long_web_013 (svea, 2026-11-04 21:01 JST, cache keep, thermal 2 before; check run, not waited for clean (thermal 2, cap policy0:2112000/3628800 policy6:2112000/4742400 , kgsl_pwrlevel 5)); Vk4e-gv-e_long_mobile_491 (svea, 2026-11-04 21:02 JST, cache keep, thermal 3 before; check run, not waited for clean (thermal 3, cap policy0:1324800/3628800 policy6:1382400/4742400 , kgsl_pwrlevel 6)); every run's model_sha256 = the artifact",
            "the four runs ran back to back in one hold, from thermal 0 to 3, the first cold: not a speed figure (the card's Long requests)",
            "full_delegation false = the decoder's 2 signatures whole on the GPU (LITERT_CL: prefill_1024 1,729 of 1,729 nodes, decode 1,654 of 1,654, 1 partition each); the vision encoder on the CPU as requested (XNNPACK 7,915 of 7,939 nodes, 49 partitions), the vision adapter (XNNPACK 20 of 26) and the embedder (XNNPACK 1 of 4) on the CPU, as the runtime log lists them",
            "delegation (Vk4e-gv-e_hyb_desktop_006): prefill_1024 1729 / 1729 on LITERT_CL (1 partition(s)); decode 1654 / 1654 on LITERT_CL (1 partition(s)); main 1 / 4 on TfLiteXNNPackDelegate (2 partition(s)); main 7915 / 7939 on TfLiteXNNPackDelegate (49 partition(s)); main 20 / 26 on TfLiteXNNPackDelegate (3 partition(s))",
            "peak_mem_mb = the highest VmHWM of these processes in MiB (phone-side samples, kB / 1024); it does not hold the GPU's memory"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "clicks_in_box_of_3": 1.0,
            "decode_tokens_per_s_p1": 18.41,
            "decode_tokens_per_s_p2": 16.87,
            "decode_tokens_per_s_p3": 16.65,
            "decode_tokens_per_s_p4": 15.55,
            "ended_at_stop_token": 4.0,
            "largest_position": 3875.0,
            "memavailable_min_kb": 2463524.0,
            "n_gen_p1": 121.0,
            "n_gen_p2": 236.0,
            "n_gen_p3": 276.0,
            "n_gen_p4": 328.0,
            "prefill_tokens_per_s_p1": 219.1,
            "prefill_tokens_per_s_p2": 219.4,
            "prefill_tokens_per_s_p3": 209.6,
            "prefill_tokens_per_s_p4": 202.2,
            "prompt_tokens_p1": 3186.0,
            "prompt_tokens_p2": 3546.0,
            "prompt_tokens_p3": 3546.0,
            "prompt_tokens_p4": 3547.0,
            "request_s_p1": 48.55,
            "request_s_p2": 54.73,
            "request_s_p3": 61.89,
            "request_s_p4": 77.4,
            "requests": 4.0,
            "tool_call_parses": 4.0,
            "whole_prompt_prefilled": 4.0
          },
          "output_match": null,
          "peak_mem_mb": 2561.8,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the long requests (decoder on the GPU, vision encoder on the CPU): the hybrid prompt as written (3,186 tokens) and three with 40 earlier-step lines (3,546-3,547 tokens)",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.0/2026-11-04/mage-vl-cowork-vision-fp16__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": 2048,
          "date": "2026-11-04",
          "decode_tokens_per_s": 20.09,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-magevl-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/Mage-VL-Cowork `Mage-VL-Cowork_vision-fp16.litertlm` at main 38bf428b: LFS sha256 1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df, 3199591888 B (anonymous Hub API tree read 2026-11-05 03:21 JST)",
            "runtime: PyPI litert-lm-api 0.19.0 Android wheel (liblitert-lm.so sha256 28fa8433febc49a99f4bd3ef8cb82c2912457cfe274d97944debeabdeb97fc78) on CPython 3.14.8, the Python API on the phone: `Benchmark(model, GPU(), prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048, cache_dir=':nocache'); _get_or_create_engine() timed; 3 x run() on one engine`",
            "protocol: a benchmark script, one process per run, the phone's state and a ready check before each run, the process stopped by the memory guard on MemAvailable",
            "the decoder alone (no vision executor): the cell = the mean of runs 2-3 of 3 (run Vk4e-gpu-t1, run svea, model_sha256 = the artifact; uncapped (thermal 1, no CPU cap, kgsl_pwrlevel 0); rested 134s since 21:03:37; MemAvailable 7882536 kB >= memmin 5000000 kB after 0s)",
            "full_delegation false = prefill_1024 1,729 of 1,729 and decode 1,654 of 1,654 on LITERT_CL, the embedder XNNPACK 1 of 4",
            "peak_mem_mb = VmHWM in MiB (phone-side samples); kgsl in MiB (the phone's GPU pages, every process)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "cpu_capped_samples": 16.0,
            "decode_tokens": 256.0,
            "decode_tokens_per_s_run1": 20.53,
            "decode_tokens_per_s_run2": 20.556,
            "decode_tokens_per_s_run3": 19.626,
            "engine_s": 6.14,
            "kgsl_page_alloc_max_mb": 4296.9,
            "memavailable_min_kb": 3390116.0,
            "prefill_tokens": 1024.0,
            "prefill_tokens_per_s_run1": 298.83,
            "prefill_tokens_per_s_run2": 290.19,
            "prefill_tokens_per_s_run3": 261.54,
            "runs": 3.0,
            "samples": 25.0,
            "thermal_first": 1.0,
            "thermal_max": 2.0,
            "ttft_s_run1": 3.475,
            "ttft_s_run2": 3.577,
            "ttft_s_run3": 3.966
          },
          "output_match": null,
          "peak_mem_mb": 1078.2,
          "prefill_tokens_per_s": 275.87,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 3771.5
        },
        "source": "data/device_runs/0.19.0/2026-11-04/mage-vl-cowork-vision-fp16__galaxy-s26.json"
      },
      {
        "device": "galaxy-s26",
        "run": {
          "accelerator": "gpu",
          "context_length": 4096,
          "date": "2026-11-04",
          "decode_tokens_per_s": 17.6,
          "delegated_ops": null,
          "env": {
            "device": "Galaxy S26 (SM-S942Q)",
            "machine_label": "galaxy-s26-magevl-devicebench",
            "os_build": "Android 16 (BP4A.251205.006)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Qualcomm SM8850 (Adreno GPU)",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/Mage-VL-Cowork `Mage-VL-Cowork_vision-fp16.litertlm` at main 38bf428b: LFS sha256 1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df, 3199591888 B (anonymous Hub API tree read 2026-11-05 03:21 JST)",
            "runtime: PyPI litert-lm-api 0.19.0 Android wheel (liblitert-lm.so sha256 28fa8433febc49a99f4bd3ef8cb82c2912457cfe274d97944debeabdeb97fc78) on CPython 3.14.8, the Python API on the phone: `Engine(model, backend=GPU(), vision_backend=CPU(), max_num_images=1, enable_benchmark=True); create_conversation(messages=[system]); send_message_async({'role': 'user', 'content': [image, text]}, max_output_tokens=1108)`; sampling: the bundle declares no sampler, so the runtime's CPU / GPU default applies (top-k 1 = greedy)",
            "protocol: a benchmark script, one process per run, the phone's state and a ready check before each run, the process stopped by the memory guard on MemAvailable",
            "the one request, the decoder on the GPU (LITERT_CL, FP16), the vision encoder on the CPU (XNNPACK, the section's FP16 metadata, vision_backend=CPU): ScreenSpot-v2 desktop_006 stretched to 1920 x 1088 + the vendor's GUI agent prompt = 2,988 tokens in, a fresh process each (a new Engine); runs: Vk4-gv-w2 (svka, 2026-11-04 17:14 JST, cache keep, thermal 1 before; uncapped (thermal 1, no CPU cap, kgsl_pwrlevel 0)); Vk4-gv-w3 (svka, 2026-11-04 17:15 JST, cache keep, thermal 1 before; uncapped (thermal 1, no CPU cap, kgsl_pwrlevel 0)); Vk4-gv-w1 (svka, 2026-11-04 17:12 JST, cache cold, thermal 0 before; clean); every run's model_sha256 = the artifact; the cell = the mean over the warm processes (each in metrics: request_s = from before Engine() to the end of the reply; engine_only_s = Engine(); conversation_s = create_conversation())",
            "ttft_ms = the runtime's time to first token, which does not hold the vision stage (first chunk less it, in metrics)",
            "full_delegation false = the decoder's 2 signatures whole on the GPU (LITERT_CL: prefill_1024 1,729 of 1,729 nodes, decode 1,654 of 1,654, 1 partition each); the vision encoder on the CPU as requested (XNNPACK 7,915 of 7,939 nodes, 49 partitions), the vision adapter (XNNPACK 20 of 26) and the embedder (XNNPACK 1 of 4) on the CPU, as the runtime log lists them",
            "kgsl_page_alloc_max_mb = the highest /sys/class/kgsl/kgsl/page_alloc (the phone's GPU pages, every process), in MiB; the card gives 4.49 GB",
            "the three processes reply with one text (sha256 a91d0c788a6f); each started from a cool phone (thermal 0 or 1, no CPU cap, the GPU at its top level); the card's 50.75 s is the mean of its rounded 49.6 and 51.9; the unrounded mean is in metrics",
            "delegation (Vk4-gv-w2): prefill_1024 1729 / 1729 on LITERT_CL (1 partition(s)); decode 1654 / 1654 on LITERT_CL (1 partition(s)); main 1 / 4 on TfLiteXNNPackDelegate (2 partition(s)); main 7915 / 7939 on TfLiteXNNPackDelegate (49 partition(s)); main 20 / 26 on TfLiteXNNPackDelegate (3 partition(s))",
            "peak_mem_mb = the highest VmHWM of these processes in MiB (phone-side samples, kB / 1024); it does not hold the GPU's memory",
            "the card: https://huggingface.co/litert-community/Mage-VL-Cowork (Performance)"
          ],
          "failure_class": null,
          "full_delegation": false,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "conversation_s_cold": 1.051,
            "conversation_s_p1": 0.274,
            "conversation_s_p2": 0.201,
            "decode_tokens_cold": 261.0,
            "decode_tokens_p1": 261.0,
            "decode_tokens_p2": 261.0,
            "decode_tokens_per_s_cold": 18.63,
            "decode_tokens_per_s_p1": 17.88,
            "decode_tokens_per_s_p2": 17.33,
            "engine_only_s_cold": 5.679,
            "engine_only_s_p1": 4.852,
            "engine_only_s_p2": 5.215,
            "first_chunk_s_cold": 31.511,
            "first_chunk_s_p1": 30.236,
            "first_chunk_s_p2": 31.916,
            "kgsl_page_alloc_max_mb": 4284.3,
            "memavailable_min_kb": 2154068.0,
            "memavailable_min_kb_cold": 2154068.0,
            "memavailable_min_kb_p1": 2575364.0,
            "memavailable_min_kb_p2": 2653812.0,
            "prefill_tokens": 2988.0,
            "prefill_tokens_per_s_cold": 274.6,
            "prefill_tokens_per_s_p1": 263.2,
            "prefill_tokens_per_s_p2": 233.8,
            "processes_cold": 1.0,
            "processes_warm": 2.0,
            "request_s_cold": 51.92,
            "request_s_p1": 49.56,
            "request_s_p2": 51.91,
            "request_s_warm_mean": 50.73,
            "skin_first_c_cold": 32.5,
            "skin_first_c_p1": 38.0,
            "skin_first_c_p2": 39.6,
            "thermal_first_cold": 0.0,
            "thermal_first_p1": 1.0,
            "thermal_first_p2": 1.0,
            "ttft_s_cold": 10.937,
            "ttft_s_p1": 11.407,
            "ttft_s_p2": 12.84,
            "vision_stage_s_cold": 20.57,
            "vision_stage_s_p1": 18.83,
            "vision_stage_s_p2": 19.08,
            "vmhwm_kb_cold": 2448864.0,
            "vmhwm_kb_p1": 2575748.0,
            "vmhwm_kb_p2": 2608944.0
          },
          "output_match": null,
          "peak_mem_mb": 2547.8,
          "prefill_tokens_per_s": 248.5,
          "provenance": "measured",
          "runs": true,
          "total_ops": null,
          "ttft_ms": 12123.2
        },
        "source": "data/device_runs/0.19.0/2026-11-04/mage-vl-cowork-vision-fp16__galaxy-s26.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": 4096,
          "date": "2026-11-04",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/Mage-VL-Cowork `Mage-VL-Cowork_vision-fp16.litertlm` at main 38bf428b: LFS sha256 1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df, 3199591888 B (anonymous Hub API tree read 2026-11-05 03:21 JST)",
            "runtime: the PyPI litert-lm 0.19.0 wheel on Python 3.14.6 (GPU = WebGPU on Metal, adapter Apple M4 Max), sampling: the bundle declares no sampler, so the runtime's CPU / GPU default applies (top-k 1 = greedy); one process per request",
            "Verdict: VERDICT vf16bw_k4 PASS both 12 in_box 8 (D16r Mac GPU 8) clause_fail 0 eos_rows 12 equal_to_gpu 0 same_point_gpu 2 peak_max 3161280448",
            "this file's speed of record is on the Galaxy S26 (its own record); peak_mem_mb = the highest /usr/bin/time -l peak memory footprint / 10^6"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "bar_in_box": 6.0,
            "ended_at_stop_token": 12.0,
            "form_failures": 0.0,
            "in_box": 8.0,
            "items": 12.0,
            "n_gen_max": 306.0,
            "n_gen_min": 235.0
          },
          "output_match": true,
          "peak_mem_mb": 3161.3,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "12 of the line's screenshots with the decoder on the GPU and the vision encoder on the CPU (vision_backend=CPU, the section's FP16 metadata), one fresh process each; bar = 6 clicks in the box",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.0/2026-11-04/mage-vl-cowork-vision-fp16__mac-studio-m4-max.json"
      },
      {
        "device": "mac-studio-m4-max",
        "run": {
          "accelerator": "gpu",
          "context_length": 4096,
          "date": "2026-11-04",
          "decode_tokens_per_s": null,
          "delegated_ops": null,
          "env": {
            "device": "Mac Studio (M4 Max, Mac16,9)",
            "machine_label": "mac-studio-m4-max",
            "os_build": "macOS 27.0 (26A428)",
            "runtime": "litert-lm",
            "runtime_version": "0.19.0",
            "soc": "Apple M4 Max",
            "vendor_sdk": null
          },
          "error": null,
          "evidence": [
            "artifact: litert-community/Mage-VL-Cowork `Mage-VL-Cowork_vision-fp16.litertlm` at main 38bf428b: LFS sha256 1e977695d8f46222653233143e95431ccf6e46bfabe2c77073abf2e8ba75c5df, 3199591888 B (anonymous Hub API tree read 2026-11-05 03:21 JST)",
            "runtime: the PyPI litert-lm 0.19.0 wheel on Python 3.14.6 (GPU = WebGPU on Metal, adapter Apple M4 Max), sampling: the bundle declares no sampler, so the runtime's CPU / GPU default applies (top-k 1 = greedy); one process per request",
            "On screens the work drew, the expected tool, target and command patterns written before any reply was read",
            "output_match = the line for these requests: the right tool on at least 80 %, a click inside the target on at least 60 %, every tool call parses and every reply ends by itself; the command patterns (2 of 5) are not in the line. The card's row: | Mac M4 Max | 10 of 11 | 4 of 6 | 11 of 11 | 11 of 11 | 2 of 5 |",
            "a first reply that calls bash is the step; after a `screenshot` reply a new conversation holds the reply and the screenshot (the vendor's two steps, the card's How to use). The same 11 through the Android app on the Galaxy S26 are the sample app's runs, not in this batch",
            "peak_mem_mb = the highest /usr/bin/time -l peak memory footprint / 10^6; full_delegation null: this path's stderr prints no delegate split"
          ],
          "failure_class": null,
          "full_delegation": null,
          "latency_p50_ms": null,
          "loads": true,
          "max_abs_diff": null,
          "max_rel_diff": null,
          "metrics": {
            "clicks_in_box_bar_pct": 60.0,
            "clicks_in_box_of_6": 4.0,
            "command_patterns_of_5": 2.0,
            "ended_at_stop_token": 11.0,
            "inferences": 20.0,
            "largest_position": 3737.0,
            "requests": 11.0,
            "right_tool": 10.0,
            "right_tool_bar_pct": 80.0,
            "tool_call_parses": 11.0
          },
          "output_match": true,
          "peak_mem_mb": 3233.1,
          "prefill_tokens_per_s": null,
          "provenance": "measured",
          "runs": true,
          "signature": "the hybrid prompt's two steps on 11 held-out requests (6 GUI, 5 shell) with the decoder on the GPU and the vision encoder on the CPU; the reply of the last inference judged",
          "total_ops": null,
          "ttft_ms": null
        },
        "source": "data/device_runs/0.19.0/2026-11-04/mage-vl-cowork-vision-fp16__mac-studio-m4-max.json"
      }
    ]
  },
  "model": {
    "family": "mage-vl",
    "id": "mage-vl-cowork-vision-fp16",
    "license": "apache-2.0",
    "source_url": "https://huggingface.co/litert-community/Mage-VL-Cowork",
    "task": "image-text-to-text"
  },
  "pitfalls": [
    "CPU path only: only the vision encoder on the CPU uses this file; its encoder was not run on a GPU. The decoder runs on the GPU and vision_backend = CPU (card L24, L313).",
    "Why fp16 on the CPU: the model metadata makes XNNPACK compute the encoder in fp16, which took the vision stage on the Galaxy S26 from 39.86 s to 18.24 s (warm, from a cool phone, these two encoders in builds with a shorter KV cache; on this file 18.83 and 19.08 s) (card L309).",
    "LayerNorm range: the LayerNorms of layers 13-23 take their input times 2^-k and their eps times 4^-k (k up to 4), because their squared differences pass fp16's largest value (65,504); the first LayerNorm takes its input times 8 and its eps times 64, because on a flat screen its variance is an fp16 subnormal, the fp16 ADD with eps returned 0, RSQRT gave inf and the attention spread the NaN to every token. Neither moves the encoder's fp32 output (bit-equal on 11 pictures) (card L311).",
    "Weights stored as fp16: the fully connected weights and biases are fp16 (a section of 613 MB against 1.22 GB), because LiteRT-LM 0.19.0's weight cache aborts the engine on the fp16 weights XNNPACK makes from fp32 ones unless the cache is off (card L311).",
    "Not the fp32 encoder's output: in fp16 on the CPU the vision output is close to fp32's, not equal (correlation 0.977-0.985 on three real screenshots, 0.53-0.55 on single-colour pictures), so this file has its own test rows (card L313).",
    "One picture size: the bundle takes one screenshot per request, stretched to 1920 x 1088 (68 x 120 patches, 2,040 image tokens), portrait phone screens included. At other sizes the original model misplaces its clicks: on the 40 desktop screenshots, 1344 x 768 was right on 3 against 21 at 1920 x 1088 (card L26, L249).",
    "fp16 positions on the GPU: fp16 holds integers exactly only up to 2,048, and a request's positions go past it (to 3,451 on the 120 test screenshots, to 3,900 with 40 earlier-step lines). The graph computes its rotary angles from position = 2048·h + l with precomputed tables for the 2048·h part (card L325).",
    "Galaxy S26: this file is the phone's form because the fp32 encoder on its GPU ends the process in the Adreno driver and the encoder on its NPU crashes the compute DSP; neither file has an NPU form (card L24, L317, L318).",
    "One after another the phone takes longer: 50.75 s from a cool phone against a median of 83.8 s; 108 of the 120 started at thermal status 3 and 118 under a CPU cap. MemAvailable fell by 4.95-5.10 GB in the three cool-phone processes, to 2.21 GB at the lowest (card Performance table, L298, L299).",
    "Everything on the Galaxy S26's CPU is slower and it ran hot: 159.7-182.7 s, right after a GPU run with no wait, at thermal status 3 and under different CPU caps, so those two do not compare the caches (card L33, L300).",
    "Pixel 11 Pro: this file ran there only in the Android app on the hybrid prompt, a vision stage of 35.6 s from thermal status 0 and 90.8 s on the next inference; it did not take the 120-screenshot test on that phone (card L301).",
    "The hybrid prompt's first reply is usually `screenshot` although the screenshot is in the request, so a step takes one or two inferences. A new conversation holding the reply and the screenshot is the second request; prompt and reply together reached 3,737 of the 4,096 tokens. On 11 requests on drawn screens the right tool came on 10 and a click inside the target on 4 of 6, on the Galaxy S26 in an Android app and on the Mac (card L132-L145).",
    "Later turns: from the second turn on LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding (this bundle's decoder takes embeddings), read in the source of 0.19.0 and measured with 0.18.1 on other models, not on these files; a new conversation per inference avoids it (card L332).",
    "Sampling: the file carries no sampler settings, so LiteRT-LM 0.19.0 samples with top-k 1 on the CPU and the GPU; every result is greedy, while the model card's own setting is temperature 0.7 and top_p 0.8 (card L216)."
  ],
  "schema_version": "1.2"
}
