mouadja02/skills / skills/agent-eval/local-ai-stack
local-ai-stack
VerifiedSet up and optimize a complete local AI infrastructure — Ollama, Open WebUI, local RAG pipelines, and private model serving. Use when running models locally for privacy, compliance, offline use, or cost, or setting up GPU-accelerated local inference and RAG.
Local AI Stack
Tier: POWERFUL Category: AI Agents Domain: Local AI Infrastructure / Self-Hosted LLM / Privacy-First AI
Overview
Build a complete, private AI stack on your own hardware — from local model serving to RAG pipelines to chat interfaces. This eliminates cloud API dependencies, keeps sensitive data local, and reduces per-token costs to zero after hardware investment. Covers Ollama for model management, Open WebUI for chat interface, and local embedding/RAG pipelines.
When to Use
- Running LLMs locally for privacy, compliance, or cost savings
- Setting up Ollama with optimized model configurations
- Deploying Open WebUI as a self-hosted ChatGPT alternative
- Building local RAG pipelines with local embeddings
- Configuring GPU acceleration (CUDA/ROCm/Metal) for inference
- Creating an air-gapped AI environment (no internet required)
- Benchmarking local models for specific use cases
- Setting up a team-shared local AI server
Quick Start: Full Local Stack in 10 Minutes
Step 1: Install Ollama
# Linux: download and inspect the official installer before execution
installer="$(mktemp)"
curl --proto '=https' --tlsv1.2 -fsSL \
https://ollama.com/install.sh -o "$installer"
${PAGER:-less} "$installer" # stop if the script is unexpected
sh "$installer"
rm -f "$installer"
# macOS: use the signed app from https://ollama.com/download/mac
# Windows (PowerShell)
winget install Ollama.Ollama
# Verify
ollama --version
Step 2: Pull Models
# Recommended starter models (by use case)
ollama pull llama3.1:8b # General purpose (fast, 4.7GB)
ollama pull qwen2.5:14b # Best quality/speed balance (8.9GB)
ollama pull codellama:13b # Code generation
ollama pull nomic-embed-text # Embeddings for RAG (274MB)
ollama pull llama3.1:70b # High quality (requires 48GB+ VRAM)
# Check downloaded models
ollama list
Step 3: Deploy Open WebUI
# Docker (recommended)
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
--name open-webui \
--restart always \
ghcr.io/open-webui/open-webui:main
# Access at http://localhost:3000
GPU Configuration
NVIDIA (CUDA)
# Check GPU
nvidia-smi
# Ollama auto-detects CUDA. Verify:
ollama run llama3.1:8b "Hello" --verbose
# Look for "gpu" in output
# For Docker with GPU passthrough:
docker run -d --gpus all -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
Apple Silicon (Metal)
# Ollama uses Metal automatically on macOS
# Check memory: models need ~60% of unified memory
# 8B model → 8GB Mac minimum
# 14B model → 16GB Mac minimum
# 70B model → 64GB Mac minimum (M2 Ultra/M3 Max)
AMD (ROCm)
# Install ROCm toolkit first, then:
HSA_OVERRIDE_GFX_VERSION=11.0.0 ollama serve
Model Selection Guide
| Use Case | Model | Size | Speed | Quality |
|---|---|---|---|---|
| Chat/General | Llama 3.1 8B | 4.7GB | ⚡⚡⚡ | ⭐⭐⭐ |
| Best balance | Qwen 2.5 14B | 8.9GB | ⚡⚡ | ⭐⭐⭐⭐ |
| Coding | CodeLlama 13B | 7.4GB | ⚡⚡ | ⭐⭐⭐⭐ |
| Coding (best) | DeepSeek Coder V2 | 8.9GB | ⚡⚡ | ⭐⭐⭐⭐⭐ |
| Reasoning | Qwen 2.5 32B | 19GB | ⚡ | ⭐⭐⭐⭐⭐ |
| Max quality | Llama 3.1 70B | 40GB | 🐌 | ⭐⭐⭐⭐⭐ |
| Embeddings | nomic-embed-text | 274MB | ⚡⚡⚡ | ⭐⭐⭐⭐ |
| Vision | LLaVA 1.6 13B | 8GB | ⚡⚡ | ⭐⭐⭐⭐ |
Local RAG Pipeline
Architecture
Documents → Chunking → Local Embeddings → Vector Store → Query
↓ ↓ ↓ ↓
LangChain nomic-embed-text ChromaDB Ollama LLM
Implementation
from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader
# 1. Load documents
loader = DirectoryLoader('./docs', glob='**/*.md')
docs = loader.load()
# 2. Chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(docs)
# 3. Embed locally (no API calls!)
embeddings = OllamaEmbeddings(model='nomic-embed-text')
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory='./chroma_db')
# 4. Query with local LLM
llm = ChatOllama(model='qwen2.5:14b', temperature=0)
retriever = vectorstore.as_retriever(search_kwargs={'k': 5})
def ask(question: str) -> str:
docs = retriever.invoke(question)
context = '\n'.join([d.page_content for d in docs])
response = llm.invoke(f"Context:\n{context}\n\nQuestion: {question}")
return response.content
Ollama API for Custom Integrations
// Ollama exposes an OpenAI-compatible API at localhost:11434
const response = await fetch('http://localhost:11434/api/chat', {
method: 'POST',
body: JSON.stringify({
model: 'qwen2.5:14b',
messages: [{ role: 'user', content: 'Hello!' }],
stream: false,
}),
});
// Or use OpenAI SDK with Ollama backend:
import OpenAI from 'openai';
const client = new OpenAI({
baseURL: 'http://localhost:11434/v1',
apiKey: 'ollama', // required but unused
});
Performance Optimization
# Ollama environment variables
OLLAMA_NUM_PARALLEL=4 # Concurrent requests
OLLAMA_MAX_LOADED_MODELS=2 # Models in VRAM simultaneously
OLLAMA_GPU_OVERHEAD=0 # Reduce if OOM
OLLAMA_FLASH_ATTENTION=1 # Enable flash attention (faster)
# Custom Modelfile for fine-tuned inference
cat > Modelfile <<EOF
FROM qwen2.5:14b
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
PARAMETER num_gpu 99
SYSTEM "You are a helpful coding assistant."
EOF
ollama create my-coder -f Modelfile
Common Pitfalls
- Model too large for VRAM — Use quantized versions (Q4_K_M) or smaller models
- Slow inference — Ensure GPU is detected; check with
ollama run --verbose - Docker can't reach Ollama — Use
--add-host=host.docker.internal:host-gateway - OOM on long contexts — Reduce
num_ctxor use a smaller model - Mixing embedding models — Always use the same embedding model for index and query
- No persistence — Mount volumes for both Ollama models and vector store data
Security for Team Deployments
- Bind Ollama to
127.0.0.1(default) or use reverse proxy with auth - Open WebUI has built-in user management with roles
- Use Docker networks to isolate services
- Enable HTTPS via reverse proxy (Caddy/nginx) for remote access
- Set
OLLAMA_ORIGINSto restrict CORS origins
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |