Skip to content

xberg-io/liter-llm / plugin/.cursor-plugin/skills/liter-llm

liter-llm

Verified

Universal LLM API client for 174 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.

FileKindSize
SKILL.mdskill md8.5 KB
index.mdmarkdown760 B
---
description: Universal LLM API client for 174 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.
license: MIT
metadata:
    author: xberg-io
    version: "0.1.0"
    repository: https://github.com/xberg-io/liter-llm
name: liter-llm
---

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:3e0dcb446dcc37cd4709da048b254902f8d7f825b553308c5d25074419895562
Source-Hash: blake3:7e83d7b11ed4cb3925b802b9dae7eb1c2b54530377fe028b1c4d7027a9765194
Schema-Version: v1
-->

# Liter-LLM Universal LLM Client

Liter-LLM is a universal LLM API client with a Rust core and native bindings for
Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C
(FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google
Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).

## Capability map

- **Modalities** — chat completions, streaming, tool/function calling,
  structured outputs, embeddings, image generation, audio (speech,
  transcription), moderation, web search, OCR, reranking, batch,
  and file operations.
- **Provider routing** — `provider/model` prefix selects the backend
  (`openai/gpt-4o`, `anthropic/claude-sonnet-4-20250514`,
  `gemini/gemini-2.0-flash`). Set a full `provider/model` as `model_hint` to choose the client provider.
- **Middleware** — response caching, rate limiting (RPM/TPM), cost tracking,
  budget enforcement (`hard` / `soft`), fallback chains, circuit-breaker
  cooldown, and background health checks.
- **Search providers (12)** and **OCR providers (4)** — first-class `search`
  and `ocr` methods routed by the same prefix convention.
- **Proxy server** — `liter-llm api` exposes an OpenAI-compatible gateway with
  22 REST endpoints, virtual API keys, budgets, and SSE streaming.
- **MCP server** — `liter-llm mcp` exposes 22 tools mirroring the proxy
  endpoints, for MCP-compatible clients (Claude Code, Claude Desktop).
- **14 language bindings** — same surface, language-native naming (snake_case
  for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).

## When to use the MCP server vs the SDK vs the proxy

| Use the … | When you want to … | Entry point |
|-----------|--------------------|-------------|
| **MCP server** | Let the agent call LLM APIs directly as tools, with no glue code | `liter-llm mcp --transport stdio` (this plugin auto-registers it) |
| **SDK / binding** | Write application code that calls LLMs in a specific language | `pip install liter-llm`, `cargo add liter-llm`, etc. |
| **Proxy** | Give many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limits | `liter-llm api --config liter-llm-proxy.toml` |

Rule of thumb: reach for the MCP server inside an agent session; the SDK when
building software; the proxy when centralizing access for multiple consumers.

## Installation

### CLI (proxy + MCP server)

```bash
# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm

# or run it without a persistent install (the CLI proxy package self-installs the binary)
pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help

# or download a prebuilt binary from the latest GitHub release:
#   https://github.com/xberg-io/liter-llm/releases/latest

# or build from source
cargo install liter-llm-cli

# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm
```

### Language bindings

| Language | Install |
|----------|---------|
| Python | `pip install liter-llm` |
| Node.js | `pnpm add @xberg-io/liter-llm` |
| Rust | `cargo add liter-llm` |
| Go | `go get github.com/xberg-io/liter-llm/packages/go/v2` |
| Ruby | `gem install liter_llm` |
| PHP | `composer require xberg-io/liter-llm` |
| C# | `dotnet add package XbergIo.LiterLlm` |
| WASM | `pnpm add @xberg-io/liter-llm-wasm` |

## Quick start (Python, async)

```python
import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(
        '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
    )
    response = await client.chat(request)
    print(response.choices[0].message.content)

asyncio.run(main())
```

## Provider routing

The prefix before `/` in the request's `model` selects the provider:

```python
ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')
```

Client construction resolves credentials for its selected provider from environment variables: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,
`GEMINI_API_KEY`, `GROQ_API_KEY`, `MISTRAL_API_KEY`, `COHERE_API_KEY`, and AWS
credentials for Bedrock. Create a separate client for each provider with matching
`model_hint` and credentials. Request prefixes change routing but reuse the client
API key; they do not select a new environment key per request.

## Configuration

Pass basic options to `create_client(...)` (`api_key`, `base_url`,
`timeout_secs`, `connect_timeout_secs`, `max_retries`, `model_hint`), or set the full surface in a
`liter-llm.toml` (SDK) / `liter-llm-proxy.toml` (proxy) — both auto-discover
from the cwd upward.

```toml
# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai/gpt-4o"
timeout_secs = 120
max_retries = 5

[cache]
max_entries = 512
ttl_seconds = 600

[budget]
global_limit = 50.0
enforcement = "hard"

[rate_limit]
rpm = 60
tpm = 100000
```

| Option | Default | Description |
|--------|---------|-------------|
| `api_key` | required | Provider key, wrapped in `SecretString` (never logged). |
| `base_url` | from registry | Override the provider base URL. |
| `model_hint` | none | Full provider/model used to select the client provider and credentials. |
| `timeout_secs` | 60 | Request timeout in seconds. |
| `connect_timeout_secs` | 10 | Connect timeout in seconds. A connection that does not open in time is retried up to `max_retries` times. Zero is an error on every target. `wasm32` ignores any other value. |
| `max_retries` | 3 | Retries on 429/5xx with exponential backoff. |
| `cache` | none | `max_entries`, `ttl_seconds`. |
| `budget` | none | `global_limit`, `model_limits`, `enforcement`. |
| `rate_limit` | none | `rpm`, `tpm`. |
| `cost_tracking` | false | Per-request cost tracking. |
| `tracing` | false | OpenTelemetry spans. |

## Proxy server

```bash
liter-llm api --config liter-llm-proxy.toml
```

22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys,
per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE
streaming, and an OpenAPI 3.1 spec at `/openapi.json`.

## MCP server

```bash
liter-llm mcp --transport stdio   # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001
```

This plugin auto-registers the stdio server via `scripts/mcp-launch.sh`. The 22
tools mirror the proxy endpoints: `chat`, `embed`, `generate_image`, `speech`,
`transcribe`, `moderate`, `rerank`, `search`, `ocr`, `list_models`; file ops
(`create_file`, `list_files`, `retrieve_file`, `delete_file`, `file_content`);
batch ops (`create_batch`, `list_batches`, `retrieve_batch`, `cancel_batch`);
and Responses API (`create_response`, `retrieve_response`, `cancel_response`).

## Common pitfalls

1. **Provider routing** — use `"provider/model"` to select a request backend. Bare
   model names fall back to the configured client provider (OpenAI by default).
2. **API keys are SecretString** — never logged or serialized. Read from env
   vars; never hardcode.
3. **Python methods are async** — `await` inside an async context.
4. **Naming conventions differ** — camelCase for TS/Node/WASM/C#/Java,
   snake_case elsewhere.
5. **Streaming chunks may have null content** — null-check
   `chunk.choices[0].delta.content` before use.
6. **Budget modes** — `hard` rejects over-budget requests; `soft` logs and
   allows.

## Additional resources

- Upstream docs: <https://docs.liter-llm.xberg.io>
- GitHub: <https://github.com/xberg-io/liter-llm>
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall