Skip to content

xberg-io/liter-llm / plugin/.cursor-plugin/skills/liter-llm

liter-llm

Verified

Universal LLM API client for 174 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.

Liter-LLM Universal LLM Client

Liter-LLM is a universal LLM API client with a Rust core and native bindings for Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C (FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).

Capability map

  • Modalities — chat completions, streaming, tool/function calling, structured outputs, embeddings, image generation, audio (speech, transcription), moderation, web search, OCR, reranking, batch, and file operations.
  • Provider routing — provider/model prefix selects the backend (openai/gpt-4o, anthropic/claude-sonnet-4-20250514, gemini/gemini-2.0-flash). Set a full provider/model as model_hint to choose the client provider.
  • Middleware — response caching, rate limiting (RPM/TPM), cost tracking, budget enforcement (hard / soft), fallback chains, circuit-breaker cooldown, and background health checks.
  • Search providers (12) and OCR providers (4) — first-class search and ocr methods routed by the same prefix convention.
  • Proxy server — liter-llm api exposes an OpenAI-compatible gateway with 22 REST endpoints, virtual API keys, budgets, and SSE streaming.
  • MCP server — liter-llm mcp exposes 22 tools mirroring the proxy endpoints, for MCP-compatible clients (Claude Code, Claude Desktop).
  • 14 language bindings — same surface, language-native naming (snake_case for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).

When to use the MCP server vs the SDK vs the proxy

Use the …When you want to …Entry point
MCP serverLet the agent call LLM APIs directly as tools, with no glue codeliter-llm mcp --transport stdio (this plugin auto-registers it)
SDK / bindingWrite application code that calls LLMs in a specific languagepip install liter-llm, cargo add liter-llm, etc.
ProxyGive many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limitsliter-llm api --config liter-llm-proxy.toml

Rule of thumb: reach for the MCP server inside an agent session; the SDK when building software; the proxy when centralizing access for multiple consumers.

Installation

CLI (proxy + MCP server)

# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm

# or run it without a persistent install (the CLI proxy package self-installs the binary)
pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help

# or download a prebuilt binary from the latest GitHub release:
#   https://github.com/xberg-io/liter-llm/releases/latest

# or build from source
cargo install liter-llm-cli

# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm

Language bindings

LanguageInstall
Pythonpip install liter-llm
Node.jspnpm add @xberg-io/liter-llm
Rustcargo add liter-llm
Gogo get github.com/xberg-io/liter-llm/packages/go/v2
Rubygem install liter_llm
PHPcomposer require xberg-io/liter-llm
C#dotnet add package XbergIo.LiterLlm
WASMpnpm add @xberg-io/liter-llm-wasm

Quick start (Python, async)

import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(
        '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
    )
    response = await client.chat(request)
    print(response.choices[0].message.content)

asyncio.run(main())

Provider routing

The prefix before / in the request's model selects the provider:

ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')

Client construction resolves credentials for its selected provider from environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, and AWS credentials for Bedrock. Create a separate client for each provider with matching model_hint and credentials. Request prefixes change routing but reuse the client API key; they do not select a new environment key per request.

Configuration

Pass basic options to create_client(...) (api_key, base_url, timeout_secs, connect_timeout_secs, max_retries, model_hint), or set the full surface in a liter-llm.toml (SDK) / liter-llm-proxy.toml (proxy) — both auto-discover from the cwd upward.

# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai/gpt-4o"
timeout_secs = 120
max_retries = 5

[cache]
max_entries = 512
ttl_seconds = 600

[budget]
global_limit = 50.0
enforcement = "hard"

[rate_limit]
rpm = 60
tpm = 100000
OptionDefaultDescription
api_keyrequiredProvider key, wrapped in SecretString (never logged).
base_urlfrom registryOverride the provider base URL.
model_hintnoneFull provider/model used to select the client provider and credentials.
timeout_secs60Request timeout in seconds.
connect_timeout_secs10Connect timeout in seconds. A connection that does not open in time is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value.
max_retries3Retries on 429/5xx with exponential backoff.
cachenonemax_entries, ttl_seconds.
budgetnoneglobal_limit, model_limits, enforcement.
rate_limitnonerpm, tpm.
cost_trackingfalsePer-request cost tracking.
tracingfalseOpenTelemetry spans.

Proxy server

liter-llm api --config liter-llm-proxy.toml

22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys, per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE streaming, and an OpenAPI 3.1 spec at /openapi.json.

MCP server

liter-llm mcp --transport stdio   # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001

This plugin auto-registers the stdio server via scripts/mcp-launch.sh. The 22 tools mirror the proxy endpoints: chat, embed, generate_image, speech, transcribe, moderate, rerank, search, ocr, list_models; file ops (create_file, list_files, retrieve_file, delete_file, file_content); batch ops (create_batch, list_batches, retrieve_batch, cancel_batch); and Responses API (create_response, retrieve_response, cancel_response).

Common pitfalls

  1. Provider routing — use "provider/model" to select a request backend. Bare model names fall back to the configured client provider (OpenAI by default).
  2. API keys are SecretString — never logged or serialized. Read from env vars; never hardcode.
  3. Python methods are async — await inside an async context.
  4. Naming conventions differ — camelCase for TS/Node/WASM/C#/Java, snake_case elsewhere.
  5. Streaming chunks may have null content — null-check chunk.choices[0].delta.content before use.
  6. Budget modes — hard rejects over-budget requests; soft logs and allows.

Additional resources

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall