Core AI model zoo

Specialization, AIModelCache & AOT compilation (the ANE-later track)

Foundation note for the ANE-later / first-run-latency track. Everything here is the official Core AI mechanism for getting a model from .aimodel to fast on-device execution, and for moving the one-time cost off the interactive path. Sources: WWDC 324 “Meet Core AI” (XJFfCVW1UZ0), 326 “Integrate on-device AI models” (gl5lD2gEhb0) — verbatim in ondevice/_wwdc{324,326}_transcript.txt; coreai-models/models/README.md, swift/.../CoreAIShared/Bundle/ModelBundle.swift, skills/.../model-authoring/references/common_issues.md, Apple docs developer.apple.com/core-ai/ + /documentation/coreai/compiling-core-ai-models-ahead-of-time.

What “specialization” is

A shipped .aimodel is a source/device-agnostic representation. To run it, the OS specializes it for the specific device + OS version. Two transforms (324/326 verbatim):

  1. a core set of compilation steps that segment, plan, and optimize compute — this is where most of the latency is;
  2. executable-artifact generation for the compute units used — these artifacts are tied to the device + OS version.

The result is cached: first load pays the cost, later loads are fast. “This process can take a significant amount of time for very large models… avoid having model specialization occur within user-interactive flows.”

This is exactly this project’s re-specialization finding: a dynamic-shape core re-specializes on every new sequence length (~60–80× per-shape compile tax). (Project memories: project_macos_speed_state, reference_wwdc_coreai_sessions; verbatim talks in ondevice/_wwdc{324,326}_transcript.txt.)

Moving the cost off the interactive path (Swift API, 324 verbatim)

// 1) Check the cache; nil => not specialized yet => gate the feature / show "preparing…"
let cache = AIModelCache.default
guard let model = try cache.model(for: modelURL, options: .default) else {
    informUser("Preparing AI features. This may take a while…"); return
}

// 2) Or specialize explicitly, ahead of first use (after asset download / on opt-in)
try await AIModel.specialize(contentsOf: modelURL)

AIModelCache also: delete unused entries, control retention policy, and share a cache across apps in one app group. SpecializationOptions configures how the model is optimized for inference (and, on macOS, the preferred compute unit — see runtime/_specialization_options.py: cpu_only(), default(), from_preferred_compute_unit_kind(ComputeUnitKind.gpu()/.ane()/...)). Article: “Managing model specialization and caching”.

Ahead-of-time (AOT) compilation — shift the compile to your dev machine

The expensive compilation step can be done ahead of time on the dev machine, producing a compiled model; the device then only finishes the (much smaller) device-specific specialization. 326 verbatim: “…do some of that compilation ahead-of-time on my development machine… there is now much less work to do and finishes significantly faster… generates one or more compiled models targeting specific device architectures… a background asset for each compiled model.”

The 4B wall — large decoders MUST ship AOT, not as a portable IR

Small decoders (≤~1–2B, e.g. MiniCPM5-1B) ship as a portable .aimodel IR and specialize on-device fine. A 4B decoder does not — verified on FastContext-1.0-4B (Qwen3-4B), iPhone 17 Pro / iOS 27:

So 4B-class GPU bundles must be AOT-compiled per device class and shipped as .aimodelc (xcrun coreai-build compile … --preferred-compute gpu --architecture h18p) — the same reason the Gemma-4B zoo bundle ships …aotc_h18p. ANE is worse at this size: the FastContext ANE bundle static-loads (31 ANE regions, ~518 s cold) but the warmup inference dies with com.apple.appleneuralengine / ANECompilerService Code=4097 (“ANE compile failed”), so the GPU AOT bundle is the only on-device path. (Source: project_fastcontext_4b_coreai on-device runs, 2026-06-27.)

Tool naming — RESOLVED (corrects the earlier “aimodelc not coreai-build” note)

Flags (full surface, from xcrun coreai-build compile --help, verified 2026-06-10)

coreai-build compile <input.aimodel> [--output <dir>] [--platform iOS|macOS|watchOS|visionOS|tvOS ...]
    [--min-deployment-version 27.0] [--preferred-compute gpu|neural-engine|none]
    [--architecture <arch> ...] [--expect-frequent-reshapes]

⚠️ expectFrequentReshapes on a FIXED-shape graph kills the AOT bundle on iOS (device-validated 2026-07-23)

The hint is not free insurance — it is a request for a reshape-tolerant specialization. Ask for it at load time on a graph whose shapes are all static and the runtime stops using the AOT specialization and compiles on device, which on iPhone 17 Pro segfaults inside the MPSGraph AICode compiler:

EXC_BAD_ACCESS (SIGSEGV) … MPSGraphAICodeCompilerDelegate getInitializedAICodeBytecodeWithPayloadPrefix:
  → Compiler_coreAI.compile(moduleBytecode:to:with:) → libODIECompiler … CompileForDelegates

No error string, no partial output — the app just dies at AIModel(contentsOf:options:).

⚠️ Architecture names track the DEVICE IDENTIFIER, not the marketing name (device-validated 2026-06-10)

The --architecture h-numbers follow the hardware device-identifier major version (iPhone18,1, Mac16,5), NOT the marketing name (“iPhone 17 Pro”, “M4 Max”):

Deployment shape (326 demo)

Bundle the models out of the app download (they add >1 GB); gate the download behind a first-run feature intro; download assets → kick off specialization (with AOT already done) → fast first inference, all off the interactive flow. Subsequent inferences use the cached specialized asset.

Status / caveats for this project (verified vs inferred)

This site is open source. Improve this page.