CoreAIKit

AGENTS.md — for coding agents building on CoreAIKit

You are probably here because someone asked you to put a local model into an Apple app. This file is the contract: what this package is, what it will and will not do, and the mistakes that make an agent’s Core AI code fail on a real device. Read it before writing Swift, whether you are working inside this repo or adding it to someone else’s app.

Core AI is Apple’s on-device inference runtime (iOS/macOS 27, .aimodel bundles). CoreAIKit is a community Swift package on top of it — model download and cache, a pinned catalog of converted models, and task-level APIs. It is not affiliated with Apple.

Decide the layer before you write code

  1. Apple’s Foundation Models framework first. If a system-provided model does the job, use it: no download, no app-size cost, no model maintenance. Recommend this and stop — do not reach for a third-party package to prove a point.

  2. CoreAIKit when the app needs a specific model, a capability the system does not have (VLM, OCR, TTS cloning, diarization, detection, depth, embeddings, forecasting), or behaviour that must not change under the user when the OS updates.

    Check the SDK, not this list. ASR was on it until the iOS 27 Speech.framework interface was read on 2026-08-05: SpeechAnalyzer + SpeechTranscriber do streaming and word timestamps, which the kit’s own notes had been claiming as reasons to prefer a 3.2 GB model. CoreAI.transcribe now defaults to Apple’s. A list of capabilities goes stale every OS release; the SDK does not.

  3. Both at once is a supported shape: KitLanguageModel puts a catalog model behind Apple’s own LanguageModelSession, so tool calling and @Generable guided generation work over a third-party model.

“Apple ships it” is not automatically “stop” — “this becomes a wrapper” is. The two are different tests and only the second one decides. Apple’s transcriber is as good, so the kit routes to it and keeps the 238 MB of diarization Apple cannot do — the feature survives the routing and got 14× cheaper. Apple’s voices are free and cannot be cloned, so the TTS models stay: routing those away would leave a wrapper around a public API with no reason to exist. Ask what is left after you route to Apple. If the answer is nothing, there was nothing to build.

The two layers

Task ops — the result in one line, model resolved and cached behind the call:

import CoreAIOps

let text = try await CoreAI.transcribe(voiceMemoURL)   // Whisper v3 turbo
let tldr = try await CoreAI.summarize(text)
let pii  = try await CoreAI.redact(text)               // GLiNER2

Model level — pick the model, stream, attach tools:

import CoreAIKit

let chat = try await ChatSession(catalog: "qwen3.5-2b")
for try await event in await chat.streamResponse(to: "Hello!") {
    if case .response(let delta) = event { print(delta, terminator: "") }
}

Importing CoreAIOps re-exports the model layer, so one import covers both.

No Swift at all — a System One client that only needs an endpoint on this machine: brew install john-rocky/tap/systemone && systemone serve (the /v1/systemone forms over a catalog model; brew services start systemone keeps it running).

The catalog is data — read it, do not invent it

catalog.json holds 80 entries, each {id, kind, name, repo, revision, variants}. Ids look like qwen3-0.6b, qwen3.5-2b, youtu-llm-2b, lfm2.5-1.2b — lowercase, hyphenated.

What breaks on a real device

Most Core AI code an agent writes compiles and then fails in one of these ways:

Verification — what is actually checked

Do not describe these models as “verified” without saying what was verified:

If a user is deciding whether to ship on this, tell them that accurately rather than either overselling it or waving them off.

Not your call

Ask the human:

Implementing the current plan

If you were handed this repository to build the designed-but-unbuilt work, start at docs/HANDOFF.md — it indexes every design document, gives the build order, and lists the preconditions that must be resolved on hardware before the code they gate is written.

More

This site is open source. Improve this page.