Skip to content

opensite-ai/opensite-skills / linux-ai-dev-stack

linux-ai-dev-stack

Unverified

Install and wire the AI toolchain on a Linux dev machine: local LLM inference (Ollama, llama.cpp, vLLM, SGLang, TabbyAPI) with VRAM-based model selection, AI coding agents and CLIs (Claude Code, Codex, aider, OpenCode) with MCP servers, ML frameworks (PyTorch/JAX, Jupyter, HuggingFace) and GPU containers, and cloud/remote GPU workflows (RunPod, Vast.ai, Lambda, remote dev over SSH and Tailscale). Use when someone wants to run local models, set up or connect AI coding agents, configure MCP servers, route agents to local or cloud models, install an ML/GPU Python stack, or offload work to a rented GPU. It runs on the machine and assumes working GPU drivers.

Linux AI Dev Stack

Mode: Configurator — runs on the machine. This is the AI-focused heart of the Linux environment suite. It installs and wires local LLM inference, AI coding agents, MCP servers, ML frameworks, and remote GPU workflows. It assumes GPU drivers and the compute stack are already working — configure those first with linux-dev-workstation.

Currency guardrail: engine, model, and package versions move fast. Every version or model name embedded below is labeled "as of 2026-07 — verify." Confirm current values with a web search at runtime before you install or quote them.

SAFETY PROTOCOL (read before running anything)

  1. Detect first. Confirm GPU vendor + VRAM before choosing an engine or model (nvidia-smi, rocm-smi, xpu-smi, system_profiler on Asahi). Never assume.
  2. Assume drivers work; do not install them here. If nvidia-smi/rocm-smi fails, stop and hand off to linux-dev-workstation for drivers/compute.
  3. Idempotent steps. Re-running a step must be safe. Check for an existing install/service before creating one.
  4. Verify after each stage. Curl the health endpoint, run ollama ps, or hit /v1/models before moving on.
  5. Confirm before destructive/irreversible actions. Get explicit user approval before deleting model caches, replacing a systemd unit, opening a firewall port, or binding a server to 0.0.0.0.
  6. Expose local servers on the LAN safely. Bind to 0.0.0.0 only on a trusted network, always set an API key, and front with a reverse proxy or Tailscale. Never put a raw model port on the public internet unauthenticated. See references/local-llm-inference.md.
  7. Treat every MCP server as untrusted code and a prompt-injection surface. Least privilege, scoped tokens, no plaintext secrets. See references/mcp-servers.md.

Skill Resources

Preflight

Detect the accelerator and read its VRAM before anything else:

nvidia-smi --query-gpu=name,memory.total --format=csv   # NVIDIA
rocm-smi --showproductname --showmeminfo vram           # AMD ROCm
xpu-smi discovery                                        # Intel Arc / XPU
# Apple Silicon (Asahi): unified memory — check `free -h` / system RAM

If the query fails, the driver/compute stack is not ready: stop and run linux-dev-workstation first. Once you have the VRAM number, pick a tier from the table below and proceed in order.

Wire-up order

Do these in sequence; each has a verify step and links to its reference.

  1. Local inference engine — install Ollama (default), or llama.cpp / vLLM / SGLang / TabbyAPI per the matrix. Verify: curl http://localhost:11434/v1/models. → references/local-llm-inference.md
  2. Model selection by VRAM tier — pull a model that fits (see tier table). Verify: ollama ps shows 100% GPU (or an acceptable split).
  3. AI coding agents + MCP — install Claude Code / Codex / aider; add MCP servers with least privilege. Verify: agent starts, claude mcp list is clean. → references/ai-coding-agents.md, references/mcp-servers.md
  4. Local/cloud model wiring — point OpenAI-shaped tools at /v1; run a LiteLLM proxy for Anthropic-API tools. Verify: a test completion returns. → references/model-wiring.md
  5. ML frameworks — create a uv/pixi env; install PyTorch/JAX for your compute platform; set up Jupyter + HuggingFace; enable GPU containers. Verify: python -c "import torch; print(torch.cuda.is_available())". → references/ml-frameworks.md
  6. Optional remote GPU — when local VRAM is not enough, rent a GPU and set up remote dev + sync + private networking. → references/remote-gpu.md

Engine selection matrix

Compact picks (full matrix in references/local-llm-inference.md; all names as of 2026-07 — verify):

EngineBest forHardware
Ollamafastest "it works", model manager, agents, auto CPU/GPU splitNVIDIA / AMD / Apple
llama.cpp / llama-servermax control, any GGUF, laptops, MoE-on-CPU, exotic HWCUDA/ROCm/Vulkan/SYCL/Metal/CPU
vLLMhigh-throughput fleet/agent serving on a capable GPUNVIDIA / AMD / TPU
SGLangagent / shared-prefix traffic (RadixAttention), day-0 modelsNVIDIA / AMD / Intel
ExLlamaV3 + TabbyAPIbest quality-per-VRAM in 24–48GB consumer VRAMNVIDIA
LM StudioGUI users / zero-config desktop serverNVIDIA / AMD / Apple

Decision flow: want it working / manage models → Ollama (or LM Studio GUI). Need control / weird HW / huge-model-on-small-VRAM → llama.cpp. Serving many users/agents → vLLM (raw QPS) or SGLang (shared prefixes / newest models). Cram the biggest best model into consumer VRAM → ExLlamaV3 + TabbyAPI (EXL3). Front several engines / team keys / cost tracking / cloud fallback → add LiteLLM.

VRAM sizing rule

Weights: VRAM_GB ≈ params_B × (bits_per_weight / 8) × 1.2, plus KV cache for your context length. At the everyday Q4_K_M (~4.8 bpw) that is ~0.6 GB per billion params before context. Shortcut: GGUF file size on disk ≈ weight VRAM; add headroom + KV cache. Long context — not weights — is what usually OOMs you, so cap --ctx-size/--max-model-len and quantize the KV cache (q8_0 ≈ half, ~no quality loss). Details in references/local-llm-inference.md.

VRAM tier → what runs well (Q4 unless noted; as of 2026-07 — verify)

GPU VRAMSweet spotExample models
8 GB3–8B @ Q4, short ctxQwen3-4B/8B, Gemma 3 4B, Llama 3.x 8B, Phi-4-mini
16 GB14B comfy / 24–32B @ Q3–Q4 tightQwen3-14B, Mistral Small, gpt-oss-20B (offload MoE)
24 GB24–32B @ Q4–Q5Qwen3-32B, Gemma 3 27B, gpt-oss-20B full
48 GB70B @ Q4 (EXL3), 32B @ Q6/Q8Llama 3.3 70B, Qwen3-32B high-quality
80GB+ / multi-GPU70B fp8/awq serving, 120B MoEgpt-oss-120B, Nemotron 3 Super, large MoE
CPU-only / big RAMMoE with active-expert offloadgpt-oss, Qwen3-MoE via -ncmoe / ik_llama.cpp

Local-model wiring gist

  • OpenAI-shaped tools (aider, Continue, OpenCode, Codex compat profiles): point base_url at the engine's /v1 and use a dummy key — Ollama http://localhost:11434/v1, llama-server :8080/v1, vLLM :8000/v1.
  • Anthropic-API tools (Claude Code) speak the Anthropic Messages API, not OpenAI Chat Completions. Run a LiteLLM proxy, name a model after a real Anthropic model, then set ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN at it. See references/model-wiring.md and templates/litellm-config.yaml.

See also

  • linux-distro-selector — choosing a distro and verifying hardware first.
  • linux-install-planner — installing Linux on the machine (runbook for a human).
  • linux-dev-workstation — GPU drivers, shell, editors, language toolchains, containers, reproducibility, and security. Run it before this skill — this skill assumes a working driver/compute stack.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 318kUpdated Claude CodeCodexUnverifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 318kUpdated Claude CodeCodexUnverifiedInstall