On-device whole-page document structuring on Core AI. A port of
opendatalab/MinerU2.5-Pro-2605-1.2B
(Apache-2.0, 1.2B) — a small, SOTA-quality document parser (OmniDocBench v1.6 95.69). Unlike a
per-prompt recognizer, MinerU2.5 does the whole page in one model: a layout pass finds the
blocks (title / text / table / formula / figure) and their reading order, then a recognition pass
reads each region into plain text, HTML tables (<table>…), or LaTeX — assembled into structured
Markdown / JSON. This is the piece the zoo’s per-region OCRs (Unlimited-OCR,
GLM-OCR) leave to a separate layout detector; MinerU folds it into the same weights.
MinerU2.5 is stock Qwen2-VL (Qwen2VLForConditionalGeneration): a Qwen2-VL ViT vision tower + a
Qwen2-0.5B text decoder, no custom code. This port is the shipped Qwen3-VL /
GLM-OCR vision idiom (the image_embeds + rope-shift static-input hook) with a Qwen2
text decode — no deepstack, no MoE, no MLA. The vision tower runs once; its image_embeds are
injected at the image-placeholder positions (V + slot, row-major over the merged grid) and the text
decodes on top. The host runs the model twice per page (layout prompt, then recognition prompts).
⬇️ Converted .aimodel bundles:
mlboydaisuke/MinerU2.5-Pro-CoreAI —
vision/ + decoder/ (768-token recognition grid, Qwen2-VL ViT fp16 + int8lin decoder, pf64
chunked prefill) + layout/ (a 1036² square recognition/detection grid for the 2-stage) +
tokenizer/. Apache-2.0.
fc1 → quick_gelu → fc2 MLP, and the standard Qwen2-VL PatchMerger
(ln_q → view(merge²) → Linear → GELU → Linear). No deepstack, no learned pos-embed — just
baked 2D-rope constants. Exported as one fp16 .aimodel; N (visual tokens) is fixed by the
export grid.tie_word_embeddings.
Separate q/k/v with bias, no q/k-norm, standard 2-norm block, silu SwiGLU, sectioned
M-RoPE [8,12,12] applied split-half (Qwen standard rotate_half — not GLM’s interleave). Driven
on the pipelined-engine S=1 contract: input_ids [1,1], position_ids, static image_embeds [N,896]
rope_shift_start + rope_shift_amount. Zero embeds + shift_start = 1<<30 → a plain Qwen2 text
decoder.KitMineruReader in a real app (ReadDoc): a page photo → structured text (headings, paragraphs,
tables) with nothing leaving the device. ~4 s/page (warm) via chunked prefill: the pf64
multifunction bundle feeds the 768 image tokens in static S=64 chunks (the engine auto-discovers
the prefill function) then decodes S=1 — the S=1 prefill (~10 s) drops to ~2.9 s, no token cut..aimodel → image_embeds
→ AOT-compiled (h16c) int8lin S=1 decoder, autoregressive greedy: title + paragraph + the full
table reconstructed row-by-row (“Quarterly Report / On-device inference shipped across all
product lines this quarter… / Whisper 809M 0.18 s/token / …”), matching the fp32 reference.
211.7 tok/s decode (int8lin S=1, AOT h16c GPU). Portrait vision vs HF: per-token cos min
0.9975.image_embeds cos 1.0002; AOT int8lin decoder teacher-forced over
706 positions — text region 24/24 exact, generated token exact.KitMineruReader (Mac; iPhone single-pass)Both grids ride the kit’s VL rope-shift runtime (VLRuntime). Pages are CLIP-normalized and
patchified (the VLImagePreprocessor non-square + .stretch/.aspectFitPad paths added for
Qwen2-VL). Two entry points:
// Single-pass (768 grid): whole page → plain text (reading order). Runs on iPhone (chunked prefill).
let reader = try await KitMineruReader(catalog: "mineru2.5-pro")
let text = try await reader.read(imageAt: documentURL)
// 2-stage (structured Markdown, tables as <table> HTML). Needs the layout bundle too.
let reader = try await KitMineruReader(
visionDir: rec.vision, decoderDir: rec.decoder,
layoutVisionDir: layout.vision, layoutDecoderDir: layout.decoder)
let markdown = try await reader.readStructured(imageAt: documentURL)
2-stage (readStructured, mirrors mineru-vl-utils two_step_extract + json2md):
VLArchitecture.mineruLayout, 37×37 = 1369
tokens) → Layout Detection: emits <|box_start|>…<|ref_start|>type… blocks. A 768 portrait grid
mis-detects here — the square high-res grid is required. Boxes are 0–1 of the square, so they map
linearly onto the original page..mineru): each block is cropped from the original
and read by type — Table Recognition: emits OTSL (<fcel>/<nl>), kept (not skipped) and
converted to <table> HTML; Formula Recognition: → LaTeX; else plain text.Verified end-to-end in the ReadDoc Mac app: an 8-block page → title/paragraphs + a 5-row <table>,
byte-identical to the reference. ~11 s load (both bundles) + ~11 s/page. The single-pass path also
runs on iPhone; the 2-stage’s 1036² layout grid is Mac-tier (two bundles, heavy).
page image → letterbox into 672×896 (aspect-fit + white pad) → CLIP-normalize
→ non-square patchify [3072, 1176] (block-major, 64×48 patches)
→ vision .aimodel → image_embeds [768, 896]
→ prompt: [ <|vision_start|>, <image>×768, <|vision_end|>, "Text Recognition:" ]
(image ids → V+slot; shift_start = img_start+768, shift_amount = 768−32 = 736)
→ decoder S=1 pipelined greedy decode → tokens → detokenize → markdown / <table>HTML / LaTeX
The full 2-stage whole-page mode (layout boxes → per-region crop → recognition → json2md) follows
opendatalab/mineru-vl-utils
(MinerUClient.two_step_extract + post_process.json2md) — prompts by type: text →
"Text Recognition:" · table → "Table Recognition:" · formula → "Formula Recognition:" ·
figure → "Image Analysis:".
conversion/export_mineru_pipelined.py
(fp16 / int8lin / int8hu; vision stays fp16).image_embeds + rope_shift_start + rope_shift_amount) and COREAI_CHUNK_THRESHOLD=1; feed the
prompt with the image placeholders rewritten to V+slot. Large decode graphs need AOT on macOS 27
(xcrun coreai-build compile … --architecture h16c --preferred-compute gpu --expect-frequent-reshapes);
h18p bundles are prepared for iPhone.knowledge/mineru-port.md.json2md reading-order
assembly) — the value over per-region OCRs. The 2-stage orchestration is host-side (Python
mineru-vl-utils is the source of truth; a Swift host is the app-integration step).MinerU2.5-2509 is AGPL-3.0 — this port uses MinerU2.5-Pro-2605 (Apache-2.0).