Skip to content

mouadja02/skills / skills/agent-eval/local-ai-stack

local-ai-stack

Verified

Set up and optimize a complete local AI infrastructure — Ollama, Open WebUI, local RAG pipelines, and private model serving. Use when running models locally for privacy, compliance, offline use, or cost, or setting up GPU-accelerated local inference and RAG.

Local AI Stack

Tier: POWERFUL Category: AI Agents Domain: Local AI Infrastructure / Self-Hosted LLM / Privacy-First AI

Overview

Build a complete, private AI stack on your own hardware — from local model serving to RAG pipelines to chat interfaces. This eliminates cloud API dependencies, keeps sensitive data local, and reduces per-token costs to zero after hardware investment. Covers Ollama for model management, Open WebUI for chat interface, and local embedding/RAG pipelines.

When to Use

  • Running LLMs locally for privacy, compliance, or cost savings
  • Setting up Ollama with optimized model configurations
  • Deploying Open WebUI as a self-hosted ChatGPT alternative
  • Building local RAG pipelines with local embeddings
  • Configuring GPU acceleration (CUDA/ROCm/Metal) for inference
  • Creating an air-gapped AI environment (no internet required)
  • Benchmarking local models for specific use cases
  • Setting up a team-shared local AI server

Quick Start: Full Local Stack in 10 Minutes

Step 1: Install Ollama

# Linux: download and inspect the official installer before execution
installer="$(mktemp)"
curl --proto '=https' --tlsv1.2 -fsSL \
  https://ollama.com/install.sh -o "$installer"
${PAGER:-less} "$installer"  # stop if the script is unexpected
sh "$installer"
rm -f "$installer"

# macOS: use the signed app from https://ollama.com/download/mac

# Windows (PowerShell)
winget install Ollama.Ollama

# Verify
ollama --version

Step 2: Pull Models

# Recommended starter models (by use case)
ollama pull llama3.1:8b          # General purpose (fast, 4.7GB)
ollama pull qwen2.5:14b          # Best quality/speed balance (8.9GB)
ollama pull codellama:13b        # Code generation
ollama pull nomic-embed-text     # Embeddings for RAG (274MB)
ollama pull llama3.1:70b         # High quality (requires 48GB+ VRAM)

# Check downloaded models
ollama list

Step 3: Deploy Open WebUI

# Docker (recommended)
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

# Access at http://localhost:3000

GPU Configuration

NVIDIA (CUDA)

# Check GPU
nvidia-smi

# Ollama auto-detects CUDA. Verify:
ollama run llama3.1:8b "Hello" --verbose
# Look for "gpu" in output

# For Docker with GPU passthrough:
docker run -d --gpus all -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Apple Silicon (Metal)

# Ollama uses Metal automatically on macOS
# Check memory: models need ~60% of unified memory
# 8B model  → 8GB Mac minimum
# 14B model → 16GB Mac minimum
# 70B model → 64GB Mac minimum (M2 Ultra/M3 Max)

AMD (ROCm)

# Install ROCm toolkit first, then:
HSA_OVERRIDE_GFX_VERSION=11.0.0 ollama serve

Model Selection Guide

Use CaseModelSizeSpeedQuality
Chat/GeneralLlama 3.1 8B4.7GB⚡⚡⚡⭐⭐⭐
Best balanceQwen 2.5 14B8.9GB⚡⚡⭐⭐⭐⭐
CodingCodeLlama 13B7.4GB⚡⚡⭐⭐⭐⭐
Coding (best)DeepSeek Coder V28.9GB⚡⚡⭐⭐⭐⭐⭐
ReasoningQwen 2.5 32B19GB⚡⭐⭐⭐⭐⭐
Max qualityLlama 3.1 70B40GB🐌⭐⭐⭐⭐⭐
Embeddingsnomic-embed-text274MB⚡⚡⚡⭐⭐⭐⭐
VisionLLaVA 1.6 13B8GB⚡⚡⭐⭐⭐⭐

Local RAG Pipeline

Architecture

Documents → Chunking → Local Embeddings → Vector Store → Query
              ↓              ↓                ↓           ↓
          LangChain    nomic-embed-text    ChromaDB    Ollama LLM

Implementation

from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader

# 1. Load documents
loader = DirectoryLoader('./docs', glob='**/*.md')
docs = loader.load()

# 2. Chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(docs)

# 3. Embed locally (no API calls!)
embeddings = OllamaEmbeddings(model='nomic-embed-text')
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory='./chroma_db')

# 4. Query with local LLM
llm = ChatOllama(model='qwen2.5:14b', temperature=0)
retriever = vectorstore.as_retriever(search_kwargs={'k': 5})

def ask(question: str) -> str:
    docs = retriever.invoke(question)
    context = '\n'.join([d.page_content for d in docs])
    response = llm.invoke(f"Context:\n{context}\n\nQuestion: {question}")
    return response.content

Ollama API for Custom Integrations

// Ollama exposes an OpenAI-compatible API at localhost:11434
const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  body: JSON.stringify({
    model: 'qwen2.5:14b',
    messages: [{ role: 'user', content: 'Hello!' }],
    stream: false,
  }),
});

// Or use OpenAI SDK with Ollama backend:
import OpenAI from 'openai';
const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama', // required but unused
});

Performance Optimization

# Ollama environment variables
OLLAMA_NUM_PARALLEL=4      # Concurrent requests
OLLAMA_MAX_LOADED_MODELS=2 # Models in VRAM simultaneously
OLLAMA_GPU_OVERHEAD=0      # Reduce if OOM
OLLAMA_FLASH_ATTENTION=1   # Enable flash attention (faster)

# Custom Modelfile for fine-tuned inference
cat > Modelfile <<EOF
FROM qwen2.5:14b
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
PARAMETER num_gpu 99
SYSTEM "You are a helpful coding assistant."
EOF
ollama create my-coder -f Modelfile

Common Pitfalls

  1. Model too large for VRAM — Use quantized versions (Q4_K_M) or smaller models
  2. Slow inference — Ensure GPU is detected; check with ollama run --verbose
  3. Docker can't reach Ollama — Use --add-host=host.docker.internal:host-gateway
  4. OOM on long contexts — Reduce num_ctx or use a smaller model
  5. Mixing embedding models — Always use the same embedding model for index and query
  6. No persistence — Mount volumes for both Ollama models and vector store data

Security for Team Deployments

  • Bind Ollama to 127.0.0.1 (default) or use reverse proxy with auth
  • Open WebUI has built-in user management with roles
  • Use Docker networks to isolate services
  • Enable HTTPS via reverse proxy (Caddy/nginx) for remote access
  • Set OLLAMA_ORIGINS to restrict CORS origins
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall