Skip to content

dpearson2699/swift-ios-skills / skills/apple-on-device-ai

apple-on-device-ai

Verified

Build private, on-device AI features on iPhone, iPad, and Mac with Foundation Models, Core ML, MLX Swift, or llama.cpp. Use when choosing an Apple-local model runtime, building an Apple Intelligence chatbot or tool-calling feature, running an LLM on Apple Silicon, converting or compressing a Python model for Core ML, or comparing on-device inference backends. For Swift Core ML loading and prediction code, use the coreml skill.

FileKindSize
SKILL.mdskill md13.4 KB
evals/evals.jsonconfig4.3 KB
references/coreml-conversion.mdmarkdown12.8 KB
references/coreml-optimization.mdmarkdown10.2 KB
references/foundation-models.mdmarkdown15.7 KB
references/mlx-swift.mdmarkdown8.2 KB

references/mlx-swift.md

View on GitHub
# MLX Swift & llama.cpp Reference

Complete reference for running open-source LLMs on Apple platforms using
MLX Swift and llama.cpp.

## Contents

- [MLX Swift](#mlx-swift)
- [llama.cpp](#llamacpp)
- [Multi-Backend Architecture](#multi-backend-architecture)
- [Built-in Apple Frameworks](#built-in-apple-frameworks)
- [Performance Best Practices](#performance-best-practices)
- [Review Checklist](#review-checklist)

## MLX Swift

Apple's ML framework for Swift. Highest sustained generation throughput on
Apple Silicon via unified memory architecture.

### Key Characteristics

- Unified memory: operations run on CPU or GPU without data transfer
- Lazy computation: operations computed only when needed
- Automatic differentiation for training
- Metal GPU acceleration
- Research-oriented but increasingly used in production

### Loading and Running LLMs

```swift
import MLX
import MLXLLM
import MLXLMCommon
import MLXLMHFAPI
import MLXLMTokenizers

let container = try await LLMModelFactory.shared.loadContainer(
    from: HubClient.default,
    using: TokenizersLoader(),
    configuration: .init(id: "mlx-community/Qwen3-4B-4bit")
)

let session = ChatSession(container)
print(try await session.respond(to: "Hello"))

// Use ModelContainer directly when you need streaming control.
let input = try await container.prepare(input: UserInput(prompt: "Hello"))
let stream = try await container.generate(
    input: input,
    parameters: GenerateParameters(temperature: 0.0)
)
for await event in stream {
    if case .chunk(let text) = event {
        print(text, terminator: "")
    }
}
```

### Recommended Models by Device

| Device | RAM | Recommended Model | Disk Size | RAM Usage |
|---|---|---|---|---|
| iPhone 12-14 | 4-6 GB | SmolLM2-135M or Qwen 2.5 0.5B | ~278 MB | ~0.3 GB |
| iPhone 15 Pro+ | 8 GB | Gemma 3n E4B 4-bit | ~2.7 GB | ~3.5 GB |
| Mac 8 GB | 8 GB | Llama 3.2 3B 4-bit | ~1.8 GB | ~3 GB |
| Mac 16 GB+ | 16 GB+ | Mistral 7B 4-bit | ~4 GB | ~6 GB |

### Memory Management Rules

1. Never exceed 60% of total RAM on iOS
2. Set MLX cache limits:
   ```swift
   Memory.cacheLimit = 512 * 1024 * 1024 // 512 MB
   ```
3. Monitor memory pressure with `Memory.snapshot()` and reduce cache under pressure
4. Unload MLX and llama.cpp models on backgrounding or memory pressure; for MLX,
   also call `Memory.clearCache()` after generation-heavy phases
5. Use "Increased Memory Limit" entitlement for larger models on iOS
6. Pre-flight memory checks before loading models
7. Validate MLX Swift and llama.cpp on physical Apple Silicon; Simulator cannot
   exercise Metal-dependent inference, memory, or performance

### Model Lifecycle Management

```swift
@Observable
class ModelManager {
    private var model: ModelContainer?
    private var generationCount = 0

    func loadModel() async throws {
        model = try await LLMModelFactory.shared.loadContainer(
            from: HubClient.default,
            using: TokenizersLoader(),
            configuration: .init(id: "mlx-community/Qwen3-4B-4bit")
        )
    }

    func unloadModel() {
        model = nil
        Memory.clearCache()
    }
}
```

Key lifecycle patterns:
- Track active generation count to distinguish "loaded but idle" from
  "generating"
- Unconditional cancellation on app backgrounding
- 5-second delayed force-unload after backgrounding
- Platform-specific memory monitoring (UIKit on iOS, DispatchSource on macOS)

### Background Handling

```swift
// iOS: Observe app lifecycle
NotificationCenter.default.addObserver(
    forName: UIApplication.didEnterBackgroundNotification,
    object: nil, queue: .main
) { _ in
    modelManager.cancelGeneration()
    Task {
        try await Task.sleep(for: .seconds(5))
        modelManager.unloadModel()
    }
}
```

## llama.cpp

C/C++ LLM inference engine. Best cross-platform support. Uses GGUF model format.

### Swift Integration (swift-llama-cpp)

```swift
import SwiftLlamaCpp

let service = LlamaService(
    modelUrl: modelURL,
    config: .init(
        batchSize: 256,
        maxTokenCount: 4096,
        useGPU: true
    )
)

let messages = [
    LlamaChatMessage(role: .system, content: "You are helpful."),
    LlamaChatMessage(role: .user, content: "Hello")
]

let stream = try await service.streamCompletion(
    of: messages,
    samplingConfig: .init(temperature: 0.8)
)
for try await token in stream {
    print(token, terminator: "")
}
```

### GGUF Quantization Levels

| Level | Quality | Size | Use Case |
|---|---|---|---|
| Q2_K | Lowest | Smallest | Extreme memory constraints |
| Q4_K_M | Good | Balanced | Mobile devices (recommended) |
| Q5_K_M | Higher | Larger | When quality matters more |
| Q8_0 | Near-original | Largest | Desktop with ample RAM |

### llama.cpp vs MLX Swift

| Aspect | llama.cpp | MLX Swift |
|---|---|---|
| Model format | GGUF | Hugging Face / MLX format |
| Platform support | Cross-platform | Apple only |
| Throughput (Apple Silicon) | Good | Best |
| Model ecosystem | Broadest | mlx-community models |
| Maturity | Very mature | Evolving |
| Memory efficiency | Excellent | Good |

## Multi-Backend Architecture

When an app needs multiple AI backends:

### Fallback Chain Pattern

```swift
func respond(to prompt: String) async throws -> String {
    // Try Foundation Models first when available (system-integrated backend)
    if SystemLanguageModel.default.isAvailable {
        return try await foundationModelsRespond(prompt)
    }

    // Fall back to MLX Swift (best throughput)
    if canLoadMLXModel() {
        return try await mlxRespond(prompt)
    }

    // Fall back to llama.cpp (broadest compatibility)
    if llamaModelAvailable() {
        return try await llamaRespond(prompt)
    }

    throw AIError.noBackendAvailable
}
```

### Architecture Guidelines

1. Create a router that checks Foundation Models availability first
2. Fall back to MLX or llama.cpp when Foundation Models is unavailable
3. Define model tiers based on device capabilities
4. Serialize all model access through a coordinator actor to prevent contention
5. Ensure tool systems work across backends (schema translation may be needed)

### Coordinator Actor

```swift
actor ModelCoordinator {
    private var activeBackend: Backend?

    func withExclusiveAccess<T>(
        _ work: () async throws -> T
    ) async rethrows -> T {
        try await work()
    }

    enum Backend {
        case foundationModels
        case mlx
        case llamaCpp
    }
}
```

## Built-in Apple Frameworks

Before reaching for custom models, consider built-in frameworks:

### Natural Language Framework

No model downloads required:

- `NLLanguageRecognizer` -- Language detection
- `NLTokenizer` -- Word, sentence, paragraph tokenization
- `NLTagger` -- Parts of speech, named entity recognition, sentiment
- `NLEmbedding` -- Word and sentence vectors, similarity search

### Vision Framework

Built-in computer vision (legacy `VN*` API; for iOS 18+ prefer modern Swift equivalents like `RecognizeTextRequest`):

- `VNRecognizeTextRequest` -- OCR
- `VNClassifyImageRequest` -- Image classification
- `VNDetectFaceRectanglesRequest` -- Face detection
- `VNDetectHumanBodyPoseRequest` -- Body pose estimation

### Create ML

Training custom classifiers directly on device or Mac:

- Image classification
- Text classification
- Tabular data models
- Sound classification

## Performance Best Practices

1. Run outside debugger for accurate benchmarks (Xcode: Cmd-Opt-R, uncheck
   "Debug Executable")
2. Use `session.prewarm()` for Foundation Models before user interaction
3. Batch Vision framework requests in a single `perform()` call
4. Use `.fast` recognition level for real-time camera processing
5. Neural Engine (Core ML) is most energy-efficient for compatible operations
6. For MLX Swift, monitor token generation speed and adjust model size if
   below acceptable thresholds

## Review Checklist

- [ ] Model size appropriate for target device RAM
- [ ] Memory pressure monitoring implemented
- [ ] Models unloaded on app backgrounding
- [ ] MLX cache limits set appropriately
- [ ] Pre-flight memory check before loading large models
- [ ] Fallback strategy when model unavailable
- [ ] All model access serialized through coordinator
- [ ] Quantization level appropriate for quality/size tradeoff
- [ ] Energy efficiency considered (Neural Engine vs GPU)
- [ ] Physical device testing (not simulator) for Metal-dependent code
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall