xberg-io/liter-llm / plugin/.cursor-plugin/skills/liter-llm
liter-llm
VerifiedUniversal LLM API client for 174 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.
Liter-LLM Universal LLM Client
Liter-LLM is a universal LLM API client with a Rust core and native bindings for Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C (FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).
Capability map
- Modalities — chat completions, streaming, tool/function calling, structured outputs, embeddings, image generation, audio (speech, transcription), moderation, web search, OCR, reranking, batch, and file operations.
- Provider routing —
provider/modelprefix selects the backend (openai/gpt-4o,anthropic/claude-sonnet-4-20250514,gemini/gemini-2.0-flash). Set a fullprovider/modelasmodel_hintto choose the client provider. - Middleware — response caching, rate limiting (RPM/TPM), cost tracking,
budget enforcement (
hard/soft), fallback chains, circuit-breaker cooldown, and background health checks. - Search providers (12) and OCR providers (4) — first-class
searchandocrmethods routed by the same prefix convention. - Proxy server —
liter-llm apiexposes an OpenAI-compatible gateway with 22 REST endpoints, virtual API keys, budgets, and SSE streaming. - MCP server —
liter-llm mcpexposes 22 tools mirroring the proxy endpoints, for MCP-compatible clients (Claude Code, Claude Desktop). - 14 language bindings — same surface, language-native naming (snake_case for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).
When to use the MCP server vs the SDK vs the proxy
| Use the … | When you want to … | Entry point |
|---|---|---|
| MCP server | Let the agent call LLM APIs directly as tools, with no glue code | liter-llm mcp --transport stdio (this plugin auto-registers it) |
| SDK / binding | Write application code that calls LLMs in a specific language | pip install liter-llm, cargo add liter-llm, etc. |
| Proxy | Give many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limits | liter-llm api --config liter-llm-proxy.toml |
Rule of thumb: reach for the MCP server inside an agent session; the SDK when building software; the proxy when centralizing access for multiple consumers.
Installation
CLI (proxy + MCP server)
# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm
# or run it without a persistent install (the CLI proxy package self-installs the binary)
pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help
# or download a prebuilt binary from the latest GitHub release:
# https://github.com/xberg-io/liter-llm/releases/latest
# or build from source
cargo install liter-llm-cli
# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm
Language bindings
| Language | Install |
|---|---|
| Python | pip install liter-llm |
| Node.js | pnpm add @xberg-io/liter-llm |
| Rust | cargo add liter-llm |
| Go | go get github.com/xberg-io/liter-llm/packages/go/v2 |
| Ruby | gem install liter_llm |
| PHP | composer require xberg-io/liter-llm |
| C# | dotnet add package XbergIo.LiterLlm |
| WASM | pnpm add @xberg-io/liter-llm-wasm |
Quick start (Python, async)
import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest
async def main() -> None:
client = create_client(api_key=os.environ["OPENAI_API_KEY"])
request = ChatCompletionRequest.from_json(
'{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
)
response = await client.chat(request)
print(response.choices[0].message.content)
asyncio.run(main())
Provider routing
The prefix before / in the request's model selects the provider:
ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')
Client construction resolves credentials for its selected provider from environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY,
GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, and AWS
credentials for Bedrock. Create a separate client for each provider with matching
model_hint and credentials. Request prefixes change routing but reuse the client
API key; they do not select a new environment key per request.
Configuration
Pass basic options to create_client(...) (api_key, base_url,
timeout_secs, connect_timeout_secs, max_retries, model_hint), or set the full surface in a
liter-llm.toml (SDK) / liter-llm-proxy.toml (proxy) — both auto-discover
from the cwd upward.
# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai/gpt-4o"
timeout_secs = 120
max_retries = 5
[cache]
max_entries = 512
ttl_seconds = 600
[budget]
global_limit = 50.0
enforcement = "hard"
[rate_limit]
rpm = 60
tpm = 100000
| Option | Default | Description |
|---|---|---|
api_key | required | Provider key, wrapped in SecretString (never logged). |
base_url | from registry | Override the provider base URL. |
model_hint | none | Full provider/model used to select the client provider and credentials. |
timeout_secs | 60 | Request timeout in seconds. |
connect_timeout_secs | 10 | Connect timeout in seconds. A connection that does not open in time is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value. |
max_retries | 3 | Retries on 429/5xx with exponential backoff. |
cache | none | max_entries, ttl_seconds. |
budget | none | global_limit, model_limits, enforcement. |
rate_limit | none | rpm, tpm. |
cost_tracking | false | Per-request cost tracking. |
tracing | false | OpenTelemetry spans. |
Proxy server
liter-llm api --config liter-llm-proxy.toml
22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys,
per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE
streaming, and an OpenAPI 3.1 spec at /openapi.json.
MCP server
liter-llm mcp --transport stdio # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001
This plugin auto-registers the stdio server via scripts/mcp-launch.sh. The 22
tools mirror the proxy endpoints: chat, embed, generate_image, speech,
transcribe, moderate, rerank, search, ocr, list_models; file ops
(create_file, list_files, retrieve_file, delete_file, file_content);
batch ops (create_batch, list_batches, retrieve_batch, cancel_batch);
and Responses API (create_response, retrieve_response, cancel_response).
Common pitfalls
- Provider routing — use
"provider/model"to select a request backend. Bare model names fall back to the configured client provider (OpenAI by default). - API keys are SecretString — never logged or serialized. Read from env vars; never hardcode.
- Python methods are async —
awaitinside an async context. - Naming conventions differ — camelCase for TS/Node/WASM/C#/Java, snake_case elsewhere.
- Streaming chunks may have null content — null-check
chunk.choices[0].delta.contentbefore use. - Budget modes —
hardrejects over-budget requests;softlogs and allows.
Additional resources
- Upstream docs: https://docs.liter-llm.xberg.io
- GitHub: https://github.com/xberg-io/liter-llm
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |