CoreAIKit

Cookbook — “I want to …”

Reverse lookup: find your task, copy the snippet. Everything runs fully on device (macOS 27 beta / iOS 27 beta) and downloads its model from the Hugging Face Hub on first use.

One product, one import: import CoreAIOps re-exports the model layer, so every snippet on this page compiles with that single import. Most entries are task ops: one line, the kit resolves and caches a catalog model behind the call, and options: .model("catalog-id") swaps it. Entries that need state — a conversation, a camera loop, a persistent index — use the model-level APIs; starting with an op and dropping down later is a refactor, not a rewrite.

The “In → out” one-liners below are also API: every op carries the same summary as CoreAI.Op.summary (with defaultModelID), so a picker or gallery UI renders straight from Op.allCases.

I want to… Call In → out
summarize text CoreAI.summarize(text) Text → short summary
pull typed values out of text CoreAI.extract(text, as: T.self) Text → typed value (@Generable)
translate CoreAI.translate(text, to: .english) Text → translation in a named language
fix grammar and typos CoreAI.proofread(text) Text → corrected text
redact PII CoreAI.redact(text) Text → text with PII replaced by labels
find names/emails/anything in text CoreAI.extractEntities(from:labels:) Text → entities by zero-shot label
chat with a local LLM, streaming ChatSession Prompt ⇄ streamed conversation
let the model call my functions KitLanguageModel + FM tools Prompt → answer via your tools
get schema-valid JSON, guaranteed guided generation Prompt → schema-valid JSON
describe a photo CoreAI.caption(photo) Image → description
ask questions about an image KitVisionModel session Image + questions ⇄ conversation
find objects in an image CoreAI.detect(in: photo) Image → labeled bounding boxes
get a depth map CoreAI.estimateDepth(in: photo) Image → relative depth map
upscale a photo ×4 CoreAI.upscale(photo) Image → ×4 upscaled image
turn a scan into Markdown CoreAI.read(documentAt: url) Document image → markdown text
search my photos by meaning CLIP ImageTextEncoder Image / text → shared vector
search document pages without OCR VisualDocumentRetriever Query + page images → ranking
rank strings by meaning CoreAI.search(query, in: docs) Query + strings → ranked matches
build on-device RAG TextEmbedder + retrieval tool Notes → grounded answers
transcribe speech CoreAI.transcribe(url) Audio file → plain-text transcript
know who said what CoreAI.transcribeMeeting(url) Audio file → speaker-attributed transcript
describe sounds, not just words CoreAI.describeAudio(url) Audio file → description of the sounds
speak text aloud CoreAI.speak(text) Text → synthesized speech (PCM)
generate music from a prompt CoreAI.compose(prompt) Prompt → generated music (PCM)
split vocals from a song CoreAI.separate(songURL) Song → vocal and instrumental stems
recognize an action in a video CoreAI.recognizeAction(videoAt:) Video clip → ranked action labels
run a model on live camera frames CameraFeed Camera → frame stream
forecast a time series CoreAI.forecast(series) Number series → 128-step forecast
ask my local model from Siri Examples/SiriAsk Siri phrase → local model answer
put my model behind Visual Intelligence Examples/VisualIntel / AskVLM System search → your model
run any .aimodel I exported GraphModel Tensors in → tensors out
preload models / show download progress CoreAI.prepare / .onDownload Models → downloaded ahead of use

First run: downloads

Every op downloads its model on first use (cached afterwards). One process-wide hook observes all of them, and prepare front-loads the download + engine load behind your loading UI so the first real call starts instantly:

CoreAI.onDownload { print("\($0.currentFile): \(Int($0.fraction * 100))%") }
try await CoreAI.prepare(.transcribe, .caption)     // any of the twenty ops

Work with text

import CoreAIOps

let tldr = try await CoreAI.summarize(article, style: .bullets)   // .concise / .oneLine
let en   = try await CoreAI.translate(review, to: .english)       // any Language("…")
let neat = try await CoreAI.proofread(draft)
let safe = try await CoreAI.redact(supportEmail)                  // "[PERSON]", "[EMAIL]", …
let ids  = try await CoreAI.extractEntities(from: email, labels: ["person", "order number"])

Typed extraction uses the same @Generable types as Apple’s FoundationModels API:

@Generable struct Order {
    @Guide(description: "Name of the ordered product") var product: String
    @Guide(description: "Number of units ordered") var quantity: Int
}
let order = try await CoreAI.extract(email, as: Order.self)

The free-text ops default to qwen3-4B — the floor at which translate holds up. When speed matters more than fidelity: options: .model("qwen3-0.6b"). redact / extractEntities are zero-shot (GLiNER2): pass any labels, not just the PII defaults.

PII redaction on device
CoreAI.redact — GLiNER2 on iPhone

Chat, tools, and guided JSON

Streaming chat with history, live stats, and a stop button — ChatSession:

import CoreAIOps

let chat = try await ChatSession(model: .qwen3_0_6B)
for try await event in chat.streamResponse(to: "What is the capital of Japan?") {
    if case .response(let delta) = event { print(delta, terminator: "") }
}

On-device chat
ChatSession — Youtu-LLM-2B on iPhone

Tool calling rides Apple’s FoundationModels API — your Tool implementations work unchanged behind a local model:

let model = try await KitLanguageModel(model: .qwen3_0_6B)
let session = LanguageModelSession(model: model, tools: [WeatherTool()])
let answer = try await session.respond(to: "What's the weather in Sapporo?")

Schema-valid JSON by construction (constrained decoding; needs engineVariant: .sequential, ~0.6B-class models):

let model = try await KitLanguageModel(model: .qwen3_0_6B, engineVariant: .sequential)
let session = LanguageModelSession(model: model)
let plan = try await session.respond(to: "Plan a day in Kyoto.", generating: TravelPlan.self)

See docs/GETTING_STARTED.md for the support matrix and Examples/FMToolDemo / Examples/GuidedDemo for runnable versions. Examples/ChatDemo is the full chat app (~150 lines, built from CoreAIKitUI components).

Understand images

import CoreAIOps

let text  = try await CoreAI.caption(imageAt: photoURL)              // or .detailed
let boxes = try await CoreAI.detect(in: photo)                       // [Detection], no NMS
let depth = try await CoreAI.estimateDepth(in: photo).cgImage()      // grayscale render
let big   = try await CoreAI.upscale(photo)                          // ×4, deterministic
Object detection Depth estimation Super-resolution
CoreAI.detect — RF-DETR CoreAI.estimateDepth — DA3 CoreAI.upscale — AdcSR ×4

A conversation about an image (follow-up questions, one session) is the model-level VLM behind the same FoundationModels API — attach the image to the prompt:

let vlm = try await KitVisionModel(catalog: "qwen3-vl-2b")
let session = LanguageModelSession(model: vlm)
let answer = try await session.respond(to: Prompt {
    "How many people are in this photo, and what are they doing?"
    Attachment(cgImage)
})

Examples/VLChat is the full image-chat app; Examples/DetectCamera runs detection at camera rate (hold a KitDetector there — the op is a one-shot convenience).

Read documents

let markdown = try await CoreAI.read(documentAt: scanURL)   // GLM-OCR, ~4 s/page

Tables survive as markup. Swap the engine per call: options: .model("mineru2.5-pro") (2-stage layout → Markdown) or "unlimited-ocr". Examples/ReadDoc compares them on your own scans.

Document OCR
CoreAI.read — GLM-OCR, ~4 s/page on iPhone

Search and rank

Rank a handful of strings right now — one op:

let hits = try await CoreAI.search("refund policy", in: paragraphs, topK: 3)

A corpus you query repeatedly wants a persistent index — hold a TextEmbedder (EmbeddingGemma, 768-d normalized) and store document vectors once:

import CoreAIOps

let embedder = try await TextEmbedder()
var docVecs: [[Float]] = []
for doc in docs { docVecs.append(try await embedder.embed(document: doc)) }  // store these
let q = try await embedder.embed(query: "refund policy")
let scores = docVecs.map { TextEmbedder.cosineSimilarity(q, $0) }  // rank by score

Photos by meaning (CLIP, one dot product per photo):

import CoreAIOps

let encoder = try await ImageTextEncoder()                      // CLIP ViT-B/32
let photoVec = try await encoder.encode(image: cgImage)         // store per photo
let query = try await encoder.encode(text: "red bike at the beach")
let score = ImageTextEncoder.cosineSimilarity(photoVec, query)

Document pages by meaning, no OCR (ColModernVBERT late interaction): see Examples/DocSearch and VisualDocumentRetriever. Full RAG — retrieval as a tool the model calls when it decides to — is Examples/DocChat (own index) and Examples/SpotlightChat (Apple’s Spotlight as the index).

Hear and speak

import CoreAIOps

let text    = try await CoreAI.transcribe(memoURL)              // Whisper v3 turbo
let meeting = try await CoreAI.transcribeMeeting(callURL)       // diarize + per-turn ASR
print(meeting.text)                                             // one line per speaker turn
let scene   = try await CoreAI.describeAudio(clipURL)           // sounds, music, setting
let voice   = try await CoreAI.speak("The build is green.")     // PCM + sample rate
let loop    = try await CoreAI.compose("warm lo-fi loop, vinyl crackle, 90 BPM")
let stems   = try await CoreAI.separate(songURL)                // vocals / instrumental

transcribe and transcribeMeeting take options: .model("parakeet-tdt-0.6b-v3") for faster English-family ASR, or "qwen3-asr-1.7b" / "nemotron-3.5-asr-streaming-0.6b". Playback / WAV writing from PCM: see Examples/OpsDemo (25-line WAV writer) and Examples/Speak (streaming synthesis into AVAudioEngine).

Speech to text Speaker diarization
CoreAI.transcribe — Whisper v3 turbo CoreAI.transcribeMeeting — Sortformer + ASR

Video and live camera

let actions = try await CoreAI.recognizeAction(videoAt: clipURL)   // V-JEPA 2
print(actions.first?.label ?? "")   // "pouring something into something"

Live pipelines are a for-await loop over CameraFeed — hold the model-level driver so nothing reloads per frame:

import CoreAIOps

let detector = try await KitDetector(catalog: "rf-detr")
for await frame in try await CameraFeed(framesPerSecond: 30).start() {
    let boxes = try await detector.detect(in: frame)
}

Examples/DetectCamera (33–39 FPS end-to-end), DepthCamera, and ActionCamera are the full apps.

Forecast numbers

let f = try await CoreAI.forecast(dailySales)     // TimesFM 2.5 — ~25 ms warm
let nextWeek = f.mean.prefix(7)                   // f.quantiles for uncertainty bands

128 steps ahead from any univariate series; Examples/Forecast charts it.

Time-series forecasting
CoreAI.forecast — TimesFM 2.5 on iPhone

Plug into the system

Escape hatches

Any stateless .aimodel you exported runs through the generic graph runner:

import CoreAIOps

let model = try await GraphModel(contentsOf: aimodelURL, computeUnits: .neuralEngine)
let out = try await model.run(["pixel_values": .float32(pixels, shape: [1, 3, 224, 224])])
This site is open source. Improve this page.