{ "artifacts": [ { "file": "VibeVoice-ASR-BitNet.litertlm", "sha256": "5ca907b0343d3e6bd9ec3dbf8aecbcc99b733633ef007a78a7e0f3502010af1b", "size_mb": 1891.155 } ], "benchmarks": [], "conversion": { "command": "load_bitnet.py (legacy -> native rename, drop the VAE decoder, ternarize the 7 LM projections per tensor exactly as VibeASR.cpp's convert_lm_to_gguf.py, strict load, save hf_native/ + lm_native/) -> export_audio_encoder.py (one signature: audio f32 [1,T,3200] -> features f32 [1,T,1536], T=225 = 30 s, acoustic + semantic encoders + projector + the -25 dBFS normaliser in-graph, wi8fc) -> EXTERNALIZE_EMBEDDER=1 CACHE=2048 PREFILL=512,128,32 export_simple_template.py lm_native … chatml_simple.jinja BMIX4_128 -> patch_tokenizer.py (speech markers as specials on 151646/7/8) -> build_bundle.py (GenericModel audio metadata, skip_mel_spectrogram_extraction, 24 kHz, frame=hop=3200; sections EMBEDDER, PREFILL_DECODE, AUDIO_ENCODER_HW, HF tokenizer) (FINDINGS Recipe)", "quantization": "LM: per-tensor ternary weights (VibeASR.cpp absmean, s = 1/mean|w|, applied at conversion; alpha 0.0297-0.0695, 36.7% zeros) stored as int4 blockwise-128 min-max — exact for a per-tensor ternary since every block is {-a,0,+a} — with int8 embedding; audio encoder int8 dynamic on FULLY_CONNECTED (wi8fc, WER-lossless), convs fp32 (HF card; FINDINGS Recipe)", "tool": "litert-torch (LM export through scripts/export_simple_template.py) + litert-torch audio-encoder export + litert-lm-builder bundle assembly (repro = hf-to-litertlm vibevoice_asr_work/: load_bitnet.py -> export_audio_encoder.py -> export_simple_template.py -> patch_tokenizer.py -> build_bundle.py)", "tool_version": "litert-torch 0.9.3 (LM export, ~/venvs/ltconv040dev) / 0.9.4 (load, encoder, bundle, eager; ~/venvs/lt094dev with torch 2.13.0, transformers 5.14.1 native vibevoice_asr, ai-edge-quantizer 0.9.0, litert-lm-builder 0.16.1); runtime gate litert-lm-api 0.16.1 (HF card Conversion notes; FINDINGS Env)" }, "cross_runtime": [], "delegation": null, "device": { "records": [ { "device": "galaxy-s26", "run": { "accelerator": "cpu", "context_length": null, "date": "2026-09-04", "decode_tokens_per_s": 35.43, "delegated_ops": 1510, "env": { "device": "Galaxy S26 (SM-S942Q)", "machine_label": "galaxy-s26-cl-pinned", "os_build": "Android 16", "runtime": "litert-lm", "runtime_version": "0.16.1", "soc": "Qualcomm SM8850", "vendor_sdk": null }, "error": null, "evidence": [ "VERBOSE: Replacing 1125 out of 1243 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 165 partitions for subgraph 0 (prefill_512).", "VERBOSE: Replacing 1125 out of 1243 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 165 partitions for subgraph 1 (prefill_128).", "VERBOSE: Replacing 1125 out of 1243 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 165 partitions for subgraph 2 (prefill_32).", "VERBOSE: Replacing 1013 out of 1132 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 170 partitions for subgraph 3 (decode).", "results block: prefill=163.01 decode=35.43 tokens/s" ], "failure_class": null, "full_delegation": false, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 755.0, "init_s": 0.52459, "prefill_tokens": 264.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 163.01, "provenance": "measured", "runs": true, "total_ops": 1573, "ttft_ms": 1650.0 }, "source": "data/device_runs/0.16.1/2026-09-04/vibevoice-asr-bitnet__galaxy-s26.json" }, { "device": "galaxy-s26", "run": { "accelerator": "gpu", "context_length": null, "date": "2026-09-04", "decode_tokens_per_s": 36.12, "delegated_ops": 1243, "env": { "device": "Galaxy S26 (SM-S942Q)", "machine_label": "galaxy-s26-cl-pinned", "os_build": "Android 16", "runtime": "litert-lm", "runtime_version": "0.16.1", "soc": "Qualcomm SM8850", "vendor_sdk": null }, "error": null, "evidence": [ "VERBOSE: Replacing 1243 out of 1243 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (prefill_512).", "VERBOSE: Replacing 1243 out of 1243 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 1 (prefill_128).", "VERBOSE: Replacing 1243 out of 1243 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 2 (prefill_32).", "VERBOSE: Replacing 1132 out of 1132 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 3 (decode).", "results block: prefill=652.54 decode=36.12 tokens/s" ], "failure_class": null, "full_delegation": true, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 11.0, "init_s": 2.83875, "prefill_tokens": 264.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 652.54, "provenance": "measured", "runs": true, "total_ops": 1243, "ttft_ms": 430.0 }, "source": "data/device_runs/0.16.1/2026-09-04/vibevoice-asr-bitnet__galaxy-s26.json" }, { "device": "mac-studio-m4-max", "run": { "accelerator": "cpu", "context_length": null, "date": "2026-09-04", "decode_tokens_per_s": 59.29, "delegated_ops": null, "env": { "device": "Mac Studio (M4 Max)", "machine_label": "mac-studio-m4-max", "os_build": null, "runtime": "litert-lm", "runtime_version": "0.16.0", "soc": null, "vendor_sdk": null }, "error": null, "evidence": [ "results block: prefill=357.33 decode=59.29 tokens/s" ], "failure_class": null, "full_delegation": null, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 256.0, "prefill_tokens": 256.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 357.33, "provenance": "measured", "runs": true, "total_ops": null, "ttft_ms": 807.9 }, "source": "data/device_runs/0.16.0/2026-09-04/vibevoice-asr-bitnet__mac-studio-m4-max.json" }, { "device": "mac-studio-m4-max", "run": { "accelerator": "gpu", "context_length": null, "date": "2026-09-04", "decode_tokens_per_s": 138.57, "delegated_ops": null, "env": { "device": "Mac Studio (M4 Max)", "machine_label": "mac-studio-m4-max", "os_build": null, "runtime": "litert-lm", "runtime_version": "0.16.0", "soc": null, "vendor_sdk": null }, "error": null, "evidence": [ "results block: prefill=2044.72 decode=138.57 tokens/s" ], "failure_class": null, "full_delegation": null, "latency_p50_ms": null, "loads": true, "max_abs_diff": null, "max_rel_diff": null, "metrics": { "decode_tokens": 256.0, "prefill_tokens": 256.0 }, "output_match": null, "peak_mem_mb": null, "prefill_tokens_per_s": 2044.72, "provenance": "measured", "runs": true, "total_ops": null, "ttft_ms": 133.8 }, "source": "data/device_runs/0.16.0/2026-09-04/vibevoice-asr-bitnet__mac-studio-m4-max.json" } ] }, "model": { "family": "vibevoice", "id": "vibevoice-asr-bitnet", "license": "mit", "source_url": "https://huggingface.co/litert-community/VibeVoice-ASR-BitNet", "task": "automatic-speech-recognition" }, "pitfalls": [ "The audio encoder must run on the CPU (audio_backend); the LM may run on CPU or GPU. On Metal/WebGPU the 30 s window exceeds the delegate's 65,535-workgroup dispatch limit ('Dispatch workgroup count X (90001) exceeds max compute workgroups per dimension'; the 20 s window still dispatches 240000, only <= ~5.5 s would fit), and on Adreno the fp16 delegate path returns an empty transcript with all 1857 nodes delegated and no error — the v3 fp16-safe RMS block did not cure it, so the remaining fp16 hazard is inside the conv stack (HF card Usage + Conversion notes; FINDINGS Gates + encoder history).", "WER (20 LibriSpeech dev-clean clips, 448 words, greedy, vendor prompt): LiteRT-LM 0.16.1 Mac CPU 2.68% = the fp32 eager reference with identical transcripts; LM on Metal 2.46%; Galaxy S26 CPU 3.12% (one clip flips 'On the' -> 'Under', an int8 dynamic-range numerics flip on Arm — a Pixel 8a produced the identical transcript); S26 GPU-LM 2.46% measured on the preceding build (identical LM sections; the re-run on the shipped build agreed on its first two clips before the phone dropped off USB). Microsoft reports 2.41% on the full test-clean set with VibeASR.cpp (HF card Correctness; FINDINGS Gates).", "Ship the 30 s window: runtime-style 10 s chunking (context reset at chunk boundaries) costs 7 words on 6 clips (4.24% vs 2.68%); sending the clip without the duration sentence works at 3.57% (the bundle's template then sends 'Please transcribe it.') — include the duration for best results (HF card; FINDINGS Gates).", "Traps that each cost real time (FINDINGS): a random lm_head shipped silently (VibeVoiceAsrForConditionalGeneration.tie_weights keys off the top-level config, so popping the tied head left it at init and every transcript was token soup — load the head explicitly and assert equal(lm_head, embed_tokens)); raw latent weights are garbage without the per-tensor ternarization (the quantization IS the model); the HF class adds vae_std*randn to the acoustic latents unconditionally (set vae_std=0, export the mean, as VibeASR.cpp does); the tokenizer has no string for the speech-marker ids (151646-151648) so they are added as specials or the generic path spells them out as BPE pieces; the runtime's AudioStaticEncoder requires an output literally named 'features'; rank-0 tensors break the GPU delegate (keep every reduction keepdim=True).", "Runtime contract verified in LiteRT-LM source (v0.16.1 tag): without an audio adapter the encoder is treated as a streaming encoder with window = T frames, audio_buffering_enabled defaults false (short clips zero-padded, valid_tokens = ceil(valid_frames / shrink)), longer clips chunked per window with context reset at boundaries; input name must be one of audio|src_inputs|segment_values; the miniaudio preprocessor resamples to audio_sample_rate_hz and, with skip-mel, frames raw PCM without input_scale (FINDINGS Traps).", "Prompt: the vendor system prompt is baked into the bundle template; the user turn renders as <|object_ref_start|> + audio embeddings + <|object_ref_end|> + newline + text; the vendor layout has no generation prompt and appending <|im_start|>assistant is measured harmless (identical transcripts) (HF card Conversion notes; FINDINGS Gates).", "Speed (LM only; the audio encoder is a fixed 1.3-1.7 s per 30 s window on the M4 Max CPU at 8 threads): Mac litert-lm benchmark 0.16.0 CPU 357 / 59.3 tok/s, GPU (Metal) 2045 / 138.6 tok/s; end-to-end RTF 0.24 CPU / 0.19 LM-on-Metal; Galaxy S26 (litert_lm_advanced_main built from the v0.16.1 tag, 264-token text prompt, one reading per cell) CPU 163 / 35.4, GPU (OpenCL) 652 / 36.1 tok/s; on the phone a 5-20 s clip transcribes in ~7-10 s wall from a cold process incl. engine load, peak private footprint 3.2 GB CPU / 2.3 GB LM-on-GPU. The Mac bench ran on out/bundle30; the LM sections are byte-identical across the encoder builds v1-v3 (FINDINGS), so the device-run records carry no artifact digest for the shipped bytes (HF card Performance; FINDINGS Speed).", "Not done at ship: the Pixel 8a re-gate on the final build and the S26 GPU-LM re-gate (both phones dropped off USB); audio encoder on GPU is parked (CPU-only) (HF card Correctness footnote; FINDINGS SHIPPED).", "License MIT inherited from microsoft/VibeVoice (LICENSE file included); changes from the original: LM ternarized and converted from fp32 safetensors, the VAE decoder (synthesis half) not included, three special tokens added, prompt template and audio-preprocessing parameters embedded as LiteRT-LM metadata (HF card License and changes)." ], "schema_version": "1.2" }