opensite-ai/opensite-skills / linux-ai-dev-stack
linux-ai-dev-stack
FlaggedInstall and wire the AI toolchain on a Linux dev machine: local LLM inference (Ollama, llama.cpp, vLLM, SGLang, TabbyAPI) with VRAM-based model selection, AI coding agents and CLIs (Claude Code, Codex, aider, OpenCode) with MCP servers, ML frameworks (PyTorch/JAX, Jupyter, HuggingFace) and GPU containers, and cloud/remote GPU workflows (RunPod, Vast.ai, Lambda, remote dev over SSH and Tailscale). Use when someone wants to run local models, set up or connect AI coding agents, configure MCP servers, route agents to local or cloud models, install an ML/GPU Python stack, or offload work to a rented GPU. It runs on the machine and assumes working GPU drivers.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.9 KB |
| agents/openai.yaml | config | 420 B |
| examples/24gb-nvidia-local-plus-cloud.md | markdown | 5.0 KB |
| references/activation.md | markdown | 2.9 KB |
| references/ai-coding-agents.md | markdown | 7.1 KB |
| references/local-llm-inference.md | markdown | 11.9 KB |
| references/mcp-servers.md | markdown | 4.5 KB |
| references/ml-frameworks.md | markdown | 5.0 KB |
| references/model-wiring.md | markdown | 4.6 KB |
| references/remote-gpu.md | markdown | 4.1 KB |
| templates/AGENTS.md | markdown | 1.8 KB |
| templates/gpu-compose.yaml | config | 1.4 KB |
| templates/litellm-config.yaml | config | 2.3 KB |
references/model-wiring.md
View on GitHub# Model Wiring — Local + Cloud
Wire AI agents to **both** cloud providers and your local engines, front them with a
LiteLLM proxy, and route tasks to the cheapest model that can do the job. Env-var
names and endpoints are **as of 2026-07 — verify.** Never keep long-lived keys in
plaintext on disk.
## 1. Cloud API keys (secure storage)
Env-var conventions:
- Anthropic: `ANTHROPIC_API_KEY`
- OpenAI: `OPENAI_API_KEY`
- Google: `GEMINI_API_KEY` / `GOOGLE_API_KEY` (or Vertex ADC +
`GOOGLE_APPLICATION_CREDENTIALS`)
- Enterprise routing: Bedrock / Vertex / Foundry via their provider env vars.
**Base-URL overrides** repoint an agent at a proxy/gateway/compatible endpoint:
- Anthropic clients (Claude Code): `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`.
- OpenAI-compatible clients: `OPENAI_BASE_URL` / `OPENAI_API_BASE`.
**Secure storage:** do **not** put keys in shell rc or a committed `.env`. Use
**1Password `op run`** (references resolved into env at launch, masked in stdout,
never on disk) or **1Password Environments** (FIFO-backed `.env`). Alternatives:
`pass`, `gopass`, `direnv`+age, Vault, OS keychain. Example placeholder values look
like `sk-...` and `${ANTHROPIC_API_KEY}` — never commit a real key.
## 2. Local endpoints
- **Ollama** is the baseline. Serve for agents/containers:
`export OLLAMA_HOST=0.0.0.0:11434 && ollama serve`. It exposes an
**OpenAI-compatible** API at `http://localhost:11434/v1`, so any OpenAI-compatible
agent (Continue, aider, OpenCode, Cline, Codex compat profiles) points there with a
dummy key. LM Studio (`:1234/v1`), llama-server (`:8080/v1`), vLLM (`:8000/v1`),
SGLang (`:30000/v1`), TabbyAPI (`:5000/v1`) do the same.
- **Protocol-mismatch gotcha:** Claude Code speaks the **Anthropic Messages API**;
Ollama/LM Studio speak **OpenAI Chat Completions**. You cannot just set
`ANTHROPIC_BASE_URL` to Ollama — you need a translation layer (LiteLLM). Note
llama-server and LM Studio *also* expose an Anthropic-compatible endpoint directly.
## 3. LiteLLM proxy — the universal glue
Run one OpenAI/Anthropic-compatible endpoint in front of any provider (cloud +
Ollama + vLLM + llama-server) with auth, logging, budgets, spend tracking, and
**model routing/fallbacks**. Two modes: the Python SDK (`from litellm import
completion` + `Router`) and the **proxy server** (virtual keys, per-project spend,
guardrails, caching, admin UI).
```bash
pipx install 'litellm[proxy]' # then author config.yaml + run `litellm`
```
**Feed Claude Code a local model:** run LiteLLM, name a model after a real Anthropic
model (so the client needs no changes), then point Claude Code at the proxy:
```bash
export ANTHROPIC_BASE_URL="http://localhost:4000"
export ANTHROPIC_AUTH_TOKEN="sk-litellm-key"
claude
```
(Anthropic notes it does not endorse/audit LiteLLM — it is a community tool.) A ready
config that maps an Anthropic-named alias to a local Ollama model plus real cloud
models is in [../templates/litellm-config.yaml](../templates/litellm-config.yaml).
Use LiteLLM to: (a) load-balance across several local GPUs/engines, (b) fall back
local → cloud, (c) give a team one keyed endpoint, (d) track cost/usage. Simpler
alternatives: **Claude Code Router** (solo task-based routing), **OpenRouter**
(hosted multi-provider gateway; ~5.5% credit fee, BYOK 0% markup).
## 4. Cost & model routing
- **Right-size the model per task.** Rough Anthropic tiering (as of 2026-07 —
verify): **Opus** (~$5/$25 per Mtok) for architecture/deep reasoning/review;
**Sonnet** for everyday coding; **Haiku** (~$1/$5, ~15× cheaper than Opus) for
search/grep/test-running/classification/simple edits.
- **Route subagents.** In Claude Code, subagents inherit the main model unless set —
on an Opus default, read-only helpers waste money. Assign Haiku/Sonnet via `model:`
in frontmatter. Pattern: **Haiku triages → Sonnet builds → Opus reviews.**
- **Offload to local.** Route autocomplete + trivial transforms to Ollama
(`qwen2.5-coder`) to cut cloud spend to zero for high-frequency low-stakes work.
- **Cut token waste.** Enable MCP tool-search/deferral (`ENABLE_TOOL_SEARCH`), keep
`AGENTS.md`/`CLAUDE.md` lean, prune unused MCP servers, use prompt caching, and
prefer plan mode before large edits. Subscription plans (Claude Pro/Max, ChatGPT
Plus/Pro for Codex) can beat metered API for heavy interactive use.
## 5. Verify the wiring
```bash
# OpenAI-shaped local endpoint
curl -s http://localhost:11434/v1/models | head
# Through the LiteLLM proxy (Anthropic-shaped, for Claude Code)
curl -s http://localhost:4000/v1/models -H "Authorization: Bearer sk-litellm-key" | head
```
A test completion returning without an auth/protocol error confirms the route.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |