dpearson2699/swift-ios-skills / skills/apple-on-device-ai
apple-on-device-ai
VerifiedBuild private, on-device AI features on iPhone, iPad, and Mac with Foundation Models, Core ML, MLX Swift, or llama.cpp. Use when choosing an Apple-local model runtime, building an Apple Intelligence chatbot or tool-calling feature, running an LLM on Apple Silicon, converting or compressing a Python model for Core ML, or comparing on-device inference backends. For Swift Core ML loading and prediction code, use the coreml skill.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.4 KB |
| evals/evals.json | config | 4.3 KB |
| references/coreml-conversion.md | markdown | 12.8 KB |
| references/coreml-optimization.md | markdown | 10.2 KB |
| references/foundation-models.md | markdown | 15.7 KB |
| references/mlx-swift.md | markdown | 8.2 KB |
references/coreml-optimization.md
View on GitHub# Core ML Optimization Reference
Complete reference for optimizing Core ML models: quantization, palettization,
pruning, performance tuning, and profiling.
Keep conversion/optimization handoffs focused on Python-side tooling and
profiling decisions. Defer Swift prediction/runtime wiring to the sibling
`coreml` skill unless the user explicitly asks for app integration code.
## Contents
- [Optimization Technique Selection](#optimization-technique-selection)
- [Post-Training Weight Quantization (Data-Free)](#post-training-weight-quantization-data-free)
- [Palettization (Weight Clustering)](#palettization-weight-clustering)
- [Pruning (Weight Sparsification)](#pruning-weight-sparsification)
- [Joint Compression (Stacking Techniques)](#joint-compression-stacking-techniques)
- [Per-Op Configuration](#per-op-configuration)
- [Quantization-Aware Training (QAT)](#quantization-aware-training-qat)
- [Swift Integration](#swift-integration)
- [MLTensor (iOS 18+)](#mltensor-ios-18)
- [Neural Engine Best Practices](#neural-engine-best-practices)
- [Model Loading Optimization](#model-loading-optimization)
- [Profiling](#profiling)
- [Common Optimization Mistakes](#common-optimization-mistakes)
## Optimization Technique Selection
| Technique | Size Reduction | Accuracy Impact | Best Compute Unit | Min OS |
|---|---|---|---|---|
| INT8 per-channel | ~4x | Low | CPU/GPU | iOS 16 |
| INT4 per-block | ~8x | Medium | GPU | iOS 18 |
| Palettization 4-bit | ~8x | Low-Medium | Neural Engine | iOS 16 |
| Palettization 2-bit | ~16x | Medium-High | Neural Engine | iOS 16 |
| W8A8 (weights+activations) | ~4x | Low | ANE (A17 Pro/M4+) | iOS 17 |
| Pruning 50% | ~2x | Low | CPU/ANE | iOS 16 |
| Pruning 75% | ~4x | Medium | CPU/ANE | iOS 16 |
## Post-Training Weight Quantization (Data-Free)
### INT8 Per-Channel Symmetric
```python
import coremltools as ct
import coremltools.optimize as cto
model = ct.models.MLModel("model.mlpackage")
op_config = cto.coreml.OpLinearQuantizerConfig(
mode="linear_symmetric", # or "linear" (asymmetric with zero-point)
weight_threshold=512, # only quantize tensors with > N elements
)
config = cto.coreml.OptimizationConfig(global_config=op_config)
compressed = cto.coreml.linear_quantize_weights(model, config=config)
compressed.save("model_int8.mlpackage")
```
### INT4 Per-Block (PyTorch, Data-Free)
```python
import coremltools.optimize as cto
config = cto.torch.quantization.PostTrainingQuantizerConfig.from_dict({
"global_config": {
"weight_dtype": "int4",
"granularity": "per_block",
"block_size": 128,
}
})
quantizer = cto.torch.quantization.PostTrainingQuantizer(model, config)
quantized_model = quantizer.compress()
```
### GPTQ Calibration-Based Quantization
```python
config = cto.torch.layerwise_compression.LayerwiseCompressorConfig.from_dict({
"global_config": {
"algorithm": "gptq",
"weight_dtype": 4,
"granularity": "per_block",
"block_size": 128,
},
"calibration_nsamples": 16,
})
compressor = cto.torch.layerwise_compression.LayerwiseCompressor(model, config)
compressed_model = compressor.compress(calibration_dataloader)
```
## Palettization (Weight Clustering)
Especially effective on the Neural Engine. 4-bit palettization typically
preserves accuracy better than 4-bit linear quantization.
### Post-Conversion Palettization
```python
op_config = cto.coreml.OpPalettizerConfig(
mode="kmeans", # "kmeans" or "uniform"
nbits=4, # {1, 2, 3, 4, 6, 8}
granularity="per_grouped_channel", # iOS 18+ for grouped
group_size=16,
)
config = cto.coreml.OptimizationConfig(global_config=op_config)
palettized = cto.coreml.palettize_weights(model, config=config)
```
### Available Bit Widths
| Bits | Unique Values | Size Reduction | Typical Quality |
|---|---|---|---|
| 8 | 256 | ~2x | Excellent |
| 6 | 64 | ~2.7x | Very good |
| 4 | 16 | ~8x | Good |
| 3 | 8 | ~10.7x | Moderate |
| 2 | 4 | ~16x | Fair |
| 1 | 2 | ~32x | Poor (binary) |
## Pruning (Weight Sparsification)
### Magnitude Pruning
```python
config = cto.coreml.OptimizationConfig(
global_config=cto.coreml.OpMagnitudePrunerConfig(
target_sparsity=0.75,
weight_threshold=2048,
)
)
pruned = cto.coreml.prune_weights(model, config=config)
```
### Threshold Pruning
```python
config = cto.coreml.OptimizationConfig(
global_config=cto.coreml.OpThresholdPrunerConfig(
threshold=1e-12,
minimum_sparsity_percentile=0.5,
)
)
pruned = cto.coreml.prune_weights(model, config=config)
```
## Joint Compression (Stacking Techniques)
Apply multiple compression techniques in sequence:
```python
# Palettize first, then prune on top
palettized = cto.coreml.palettize_weights(model, pal_config)
final = cto.coreml.prune_weights(
palettized, prune_config, joint_compression=True
)
```
## Per-Op Configuration
Fine-grained control over which operations get compressed:
```python
config = cto.coreml.OptimizationConfig(
global_config=global_op_config,
op_type_configs={
"linear": linear_config,
"conv": conv_config,
},
op_name_configs={
"embedding_layer": None, # None = skip compression
},
)
```
## Quantization-Aware Training (QAT)
Train with quantization in the loop for best accuracy:
```python
from coremltools.optimize.torch.quantization import (
LinearQuantizer, LinearQuantizerConfig, ModuleLinearQuantizerConfig
)
config = LinearQuantizerConfig(
global_config=ModuleLinearQuantizerConfig(
quantization_scheme="symmetric",
milestones=[0, 1000, 1000, 0],
)
)
quantizer = LinearQuantizer(model, config)
quantizer.prepare(example_inputs=[1, 3, 224, 224], inplace=True)
# Training loop
for inputs, labels in data:
output = model(inputs)
loss = loss_fn(output, labels)
loss.backward()
optimizer.step()
quantizer.step()
model = quantizer.finalize(inplace=True)
```
## Swift Integration
### Loading Models
```swift
// From Xcode-compiled model (auto-generated class)
let model = try MyImageClassifier(configuration: MLModelConfiguration())
// From URL at runtime
let config = MLModelConfiguration()
config.computeUnits = .all
let model = try MLModel(contentsOf: modelURL, configuration: config)
// From pre-compiled model (.mlmodelc) for faster loading
let compiledURL = try MLModel.compileModel(at: sourceModelURL)
let model = try MLModel(contentsOf: compiledURL)
```
### MLModelConfiguration
```swift
let config = MLModelConfiguration()
config.computeUnits = .all
config.allowLowPrecisionAccumulationOnGPU = true
// config.functionName = "adapter_1" // For multifunction models (iOS 18+)
```
### Synchronous Prediction
```swift
let input = MyModelInput(image: pixelBuffer)
let output = try model.prediction(input: input)
let label = output.classLabel
```
### Async Prediction (iOS 17+)
```swift
let output = try await model.prediction(input: input)
```
Thread-safe, supports Task cancellation, integrates with Swift concurrency.
~60% faster than synchronous for batch workloads.
### Batch Prediction
```swift
let batchInputs: [MyModelInput] = images.map { MyModelInput(image: $0) }
let batchOutputs = try model.predictions(inputs: batchInputs)
```
### MLFeatureProvider
```swift
let features = try MLDictionaryFeatureProvider(dictionary: [
"input": MLFeatureValue(pixelBuffer: pixelBuffer),
"threshold": MLFeatureValue(double: 0.5),
])
let output = try model.prediction(from: features)
```
### Vision Framework Integration
```swift
import Vision
import CoreML
let vnModel = try VNCoreMLModel(for: MyDetector().model)
let request = VNCoreMLRequest(model: vnModel) { request, error in
guard let results = request.results as? [VNClassificationObservation] else { return }
let topResult = results.first
print("\(topResult?.identifier ?? ""): \(topResult?.confidence ?? 0)")
}
let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])
```
### Natural Language Integration
```swift
import NaturalLanguage
let nlModel = try NLModel(mlModel: SentimentClassifier().model)
let sentiment = nlModel.predictedLabel(for: "Great product!")
```
## MLTensor (iOS 18+)
Swift type for multidimensional array operations:
```swift
import CoreML
let tensor = MLTensor([1.0, 2.0, 3.0, 4.0])
let reshaped = tensor.reshaped(to: [2, 2])
let result = tensor.softmax()
let matmulResult = tensorA.matmul(tensorB)
```
## Neural Engine Best Practices
1. Use EnumeratedShapes instead of RangeDim for ANE optimization
2. Avoid unsupported ANE ops -- they cause fallback to CPU/GPU with transfer
overhead
3. Use palettization (4-bit or 6-bit) for best ANE memory/latency gains
4. W8A8 quantization on A17 Pro / M4+ enables optimized INT8 compute on ANE
## Model Loading Optimization
1. Pre-compile models -- use `.mlmodelc` for instant loading after first
compilation
2. Cache compiled models to a fixed location after `MLModel.compileModel(at:)`
3. Use `bisect_model()` for very large models that are slow to load
4. Use `MLComputePlan` (iOS 17.4+) for programmatic profiling
## Profiling
1. **Xcode Performance tab** -- open .mlpackage in Xcode to see load time,
prediction time, per-op compute unit assignment
2. **Core ML Instrument** in Instruments app -- runtime profiling
3. **MLComputePlan API** -- programmatic access to profiling data
4. **coremltools debugging** -- MLModelValidator, MLModelComparator,
MLModelInspector, MLModelBenchmarker
### Reshape Frequency Hint
```python
model = ct.models.MLModel("model.mlpackage",
optimization_hints={
"reshapeFrequency": ct.ReshapeFrequency.Infrequent
})
```
## Common Optimization Mistakes
1. **Applying quantization without checking accuracy.** Always validate after
compression. Use MLModelComparator to compare outputs.
2. **Ignoring weight_threshold.** Small tensors (< 512 elements) should not be
quantized -- overhead outweighs the benefit.
3. **Using synchronous predictions in async contexts.** Use async prediction
(iOS 17+) in Swift concurrency code.
4. **Not pre-compiling models.** First load triggers device-specific
compilation, which can be slow.
5. **Ignoring compute_units configuration.** Default `.all` is correct for
production. `.cpuOnly` is for debugging only.
6. **Not testing on physical devices.** Simulator does not support Metal GPU or
Neural Engine.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |