CoreAIKit

Getting Started with CoreAIKit

Build an on-device LLM app on Apple’s Core AI framework in about ten lines. No Python, no model conversion — starter models are hosted on the Hugging Face Hub and download in-app.

Requirements

• macOS 27 beta or iOS 27 beta (real device — the CoreAI framework is not in the iOS Simulator SDK), Xcode 27 beta
• ~1–5 GB of disk per model (cached after first download)

1. Add the package

In Xcode: File ▸ Add Package Dependencies… ▸ https://github.com/john-rocky/coreai-kit, then add the CoreAIKit product to your target. Or in Package.swift:

.package(url: "https://github.com/john-rocky/coreai-kit", branch: "main"),
// target dependency:
.product(name: "CoreAIKit", package: "coreai-kit"),

2. The quick path: task ops

CoreAIOps (a separate product in the same package) is the task-level layer: state the task and the kit resolves — and caches — a catalog model behind it, like a Vision framework request. Every op takes options: .model("catalog-id") to override the model per call, and because the ops ride the model-level APIs below, outgrowing one is a refactor, not a rewrite. Adding this one product is enough — import CoreAIOps re-exports the model layer (ChatSession, KitDetector, TextEmbedder, …), so no snippet in this guide needs a second product or import.

import CoreAIOps

let text = try await CoreAI.transcribe(voiceMemoURL)            // speech → text
let tldr = try await CoreAI.summarize(text, style: .bullets)    // text → summary
let todo = try await CoreAI.extract(text, as: ActionItems.self) // text → @Generable type
let ja   = try await CoreAI.translate(text, to: .japanese)
let safe = try await CoreAI.redact(text)                        // "[PERSON]", "[EMAIL]", …

let cap  = try await CoreAI.caption(photo)                      // image → description
let dets = try await CoreAI.detect(in: photo)                   // image → [Detection]
let page = try await CoreAI.read(documentAt: scanURL)           // document → markdown
let hits = try await CoreAI.search(query, in: paragraphs)       // semantic ranking
let wave = try await CoreAI.speak(reply)                        // text → speech (PCM)

Twenty ops in all — also proofread, extractEntities, transcribeMeeting (who said what), describeAudio, compose (text → music), separate (vocals / instrumental), upscale, estimateDepth, recognizeAction, and forecast (time series). The Cookbook maps every “I want to …” to its snippet; Examples/OpsDemo runs the core pipeline end to end.

First use of an op downloads its model (cached afterwards). Watch and front-load that from one place — no progress parameter on twenty ops:

CoreAI.onDownload { print("\($0.currentFile): \(Int($0.fraction * 100))%") }
try await CoreAI.prepare(.transcribe, .summarize)   // behind your loading UI;
                                                    // the first real call starts instantly

The rest of this guide is the model-level layer: you pick the model, hold the session, and stream.

3. Chat

import CoreAIKit

let chat = try await ChatSession(model: .qwen3_0_6B)   // downloads on first use
for try await event in chat.streamResponse(to: "What is the capital of Japan?") {
    switch event {
    case .response(let delta): print(delta, terminator: "")
    case .thinking(let delta): break   // qwen3 reasoning, if you want to show it
    case .stats(let stats):    break   // live TTFT / tok/s / token counts
    case .complete(let message): print("\nDone: \(message.content.count) chars")
    }
}

One-shot convenience without events:

let answer = try await chat.respond(to: "And its population?")  // history carries over

ChatSession keeps the conversation history (chat.history), re-rendering it through the bundle’s own chat template every turn. chat.reset() starts a fresh conversation without reloading the model; chat.cancelGeneration() is your stop button (the stream completes with the partial message).

Configuration

var config = ChatSession.Configuration()
config.temperature = nil            // greedy decoding (default 0.7)
config.maxResponseTokens = 1024     // default 2048
config.systemPrompt = "You are a terse assistant."
let chat = try await ChatSession(model: .qwen3_0_6B, configuration: config)

Download progress

let chat = try await ChatSession(model: .qwen3_4B) { progress in
    print("\(progress.currentFile): \(Int(progress.fraction * 100))%")
}

Models cache under Application Support/CoreAIKit/Models. Manage them with ModelStore (downloadedModels(), delete(_:), or a custom ModelStore(directory:)).

4. Starter models

Model	ModelID	macOS	iOS	Notes
Qwen3 0.6B	.qwen3_0_6B	✓	✓	smallest, thinking model
Qwen3 4B	.qwen3_4B	✓	✓	thinking model
Mistral 7B v0.3	.mistral_7B	✓	–
Gemma 3 4B	.gemma3_4B	✓	–

Any other Hugging Face repo with the same bundle layout works: ModelID("org/name", path: "macos").

The live list (with download sizes) is also available as a remote catalog — new models reach your picker without a package update:

let catalog = await ModelCatalog.load()        // falls back to a built-in snapshot offline
for entry in catalog.available(.chat) {
    print(entry.name, entry.variant?.sizeMB ?? 0, "MB")   // entry.modelID feeds ChatSession
}

Tip: call try await chat.prewarm() right after init (while your UI still shows a loading state) — it compiles the sampler graph so the first turn starts instantly.

5. Use a local bundle

Already have a bundle exported with Apple’s recipes (coreai.llm.export)?

let chat = try await ChatSession(bundleAt: URL(fileURLWithPath: "/path/to/bundle"))

The bundle directory holds metadata.json, a *.aimodel/, and a tokenizer/.

6. Run the example app

cd Examples/ChatDemo
xcodegen generate          # brew install xcodegen (once)
open ChatDemo.xcodeproj

Signing: the example projects don’t hard-code a team. Either pick yours once in Xcode’s Signing & Capabilities tab, or export DEVELOPMENT_TEAM=XXXXXXXXXX (your team id) before xcodegen generate and every example picks it up.

Computer vision: CLIP in five lines

CoreAIKitVision is a separate product — CV apps don’t link any LLM runtime.

import CoreAIKitVision

let encoder = try await ImageTextEncoder()   // downloads CLIP ViT-B/32 (~290 MB) on first use
let imageVec = try await encoder.encode(image: cgImage)        // preprocessing included
let textVec  = try await encoder.encode(text: "red bike at the beach")
let score = ImageTextEncoder.cosineSimilarity(imageVec, textVec)

Embeddings are L2-normalized 512-d vectors; ranking a photo library is one dot product per photo (see Examples/PhotoSearch).

Monocular depth is two lines (Examples/DepthCamera runs it live):

let depth = try await DepthEstimator()       // downloads Depth Anything 3 small (~100 MB)
let map = try await depth.estimateDepth(for: cgImage)
imageView.image = map.cgImage()              // min-max-normalized grayscale

Live camera pipelines are a for-await loop:

for await frame in try await CameraFeed(framesPerSecond: 5).start() {
    let map = try await depth.estimateDepth(for: frame)
}
// The app needs NSCameraUsageDescription; only the newest frame is buffered, so slow
// consumers skip frames instead of lagging.

Any other stateless .aimodel graph runs through the generic GraphModel:

let model = try await GraphModel(contentsOf: aimodelURL, computeUnits: .neuralEngine)
let out = try await model.run(["pixel_values": .float32(pixels, shape: [1, 3, 224, 224])])
let depth = out["depth"]!.floats()

SwiftUI components

CoreAIKitUI ships the pieces every model app rebuilds — a catalog-driven ModelPickerBar (selection + load button + status + download progress), a ChatTranscriptView (bubbles, thinking disclosure, auto-scroll), and a StatsBar (load / TTFT / tok/s / memory). Examples/ChatDemo is built from them; its whole UI fits in ~60 lines.

Text embeddings and local RAG

CoreAIKitEmbeddings gives you on-device text embeddings (EmbeddingGemma, normalized 768-d, multilingual):

import CoreAIKitEmbeddings

let embedder = try await TextEmbedder()        // downloads EmbeddingGemma (~590 MB)
let doc = try await embedder.embed(document: "Tokyo is the capital of Japan.")
let query = try await embedder.embed(query: "what is the capital of Japan")
let score = TextEmbedder.cosineSimilarity(doc, query)

The asymmetric retrieval prompts (query vs document) are applied automatically. Combine with KitLanguageModel and a retrieval Tool for complete on-device RAG — the model decides when to search, the framework executes it, and the answer is grounded on your documents. Examples/DocChat is the whole loop in ~150 lines:

swift run -c release DocChat ~/notes "What do my notes say about the bike trip?"

Tool calling with LanguageModelSession

KitLanguageModel puts a Core AI bundle behind Apple’s FoundationModels session API — including tool calling, which Apple’s own CoreAILanguageModel adapter does not implement.

import CoreAIKit
import FoundationModels

struct WeatherTool: Tool {
    let name = "get_weather"
    let description = "Get the current weather for a city."

    @Generable
    struct Arguments {
        @Guide(description: "Name of the city, in English")
        var city: String
    }

    func call(arguments: Arguments) async throws -> String {
        "Sunny, 24 degrees Celsius in \(arguments.city)."
    }
}

let model = try await KitLanguageModel(model: .qwen3_0_6B)
let session = LanguageModelSession(model: model, tools: [WeatherTool()])
let answer = try await session.respond(to: "What's the weather in Sapporo right now?")

The framework owns the conversation transcript (persist session.transcript and restore it to continue a conversation later), executes tool calls, and replays results into the next model turn. Retrieval-augmented flows are “define a retrieval tool” — the model decides when to search.

Support matrix

Model	ChatSession	thinking	FM chat	FM tools
Qwen3 0.6B / 4B	✓	<think>	✓	✓ (Hermes ChatML)
Mistral 7B v0.3	✓	–	✓	not yet (dialect)
Gemma 3 4B	✓	–	✓	not yet (dialect)
gpt-oss (local bundle)	✓ (harmony parsed)	analysis channel	–	not yet

Known beta caveats

• The engine ignores a consumer break and keeps generating to maximumResponseTokens in the background; EOS-ended turns therefore reset the KV cache on the next turn (correctness first). Capped turns (no EOS) hit the append-only KV fast path and report cachedTokenCount in usage. Set COREAI_KIT_DEBUG=1 to watch the decisions.
• Guided generation needs the sequential engine (per-step logits); on the default pipelined engine schema requests throw unsupportedCapability.
• Usage/metadata events are sent once at end of turn (an upfront usage event materializes an empty transcript entry on tool turns in the current beta).

Guided generation (schema-valid JSON by construction)

Constrained decoding masks the model’s per-step logits through a JSON schema’s grammar (xgrammar bitmask) before sampling — the output cannot violate the schema, so there are no parse retries. It needs per-step logits, which the default GPU-pipelined engine does not expose: load with engineVariant: .sequential (slower decode; fine for short structured output). Constrained turns can’t think — the grammar starts at the JSON.

With ChatSession (JSON schema string in, Codable out):

struct CityFacts: Codable { let name: String; let country: String }

var config = ChatSession.Configuration()
config.engineVariant = .sequential
let chat = try await ChatSession(model: .qwen3_0_6B, configuration: config)

let facts = try await chat.respond(
    to: "Give facts about the capital of Japan.",
    generating: CityFacts.self,
    schema: """
        {"type": "object",
         "properties": {"name": {"type": "string"}, "country": {"type": "string"}},
         "required": ["name", "country"]}
        """)

respondJSON(to:schema:) returns the raw JSON text, and streamGuidedResponse streams it token by token. With FoundationModels, @Generable types work end to end:

let model = try await KitLanguageModel(model: .qwen3_0_6B, engineVariant: .sequential)
let session = LanguageModelSession(model: model)
let plan = try await session.respond(to: "Plan a trip.", generating: TravelPlan.self)

See Examples/GuidedDemo for both paths runnable from the command line.

Performance notes

• Benchmark in Release. The engine’s per-token host work is ~3x slower in Debug builds.
• First load compiles on-device (one-time per model; cached afterwards). Subsequent loads are seconds.
• stats.tokensPerSecond is a 32-token rolling window — the engine bursts at decode start, so a cumulative average would over-read on short replies.
• The engine keeps its KV cache across turns and prefills only the tokens beyond the longest shared prefix with the previous turn (implicit prefix caching + reset(to:) rewind), so multi-turn TTFT stays flat. Engines that can’t rewind mid-sequence (recurrent/SSM hybrids) fall back to a full re-prefill on divergence — lossless either way. reset() clears the conversation and the cache.
• The first turn whose prefill length exceeds every earlier one pays a one-time engine cost at that length (shape specialization — a 2-4 s TTFT spike on device). On dynamic-shape (GPU) engines, warm it away during loading with prewarm(prefillLength: 256) (any length ≥ your typical prompts; covers everything shorter). Static-shape (ANE) engines skip the warm internally, and it is wasted per-token work on S=1 zoo ports (catalog hint “pipelined”) — leave it nil there.

Roadmap

• Tool dialects for more families (gpt-oss harmony, gemma, mistral)
• Guided generation on the pipelined engine (needs an engine logits path)
• More typed CV pipelines (depth, detection) over GraphModel

This site is open source. Improve this page.