Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/langfuse

langfuse

Verified

FileKindSize
SKILL.mdskill md8.4 KB
<!--
CIANCHOSAINT wholesale-copy of cianfhoghlaim/cianfhoghlaim @ main branch.

Original: cianfhoghlaim/cianfhoghlaim (per the openspec/changes/cianchosaint-repo-bootstrap-v2/specs/cianchosaint-bootstrap-v2/spec.md).
Migrated to cianchosaint: 2026-08-23
Licence: BUSL-1.1 (per LICENSE.md)

Wholesale-copied from cianfhoghlaim/cianfhoghlaim@langfuse/SKILL.md.
-->
---
name: langfuse
description: Expert assistance for LLM observability with Langfuse (Python SDK v4, JS/TS SDK v5 on platform v3.125+). Use when users need OpenTelemetry-native tracing, prompt management, evaluation, scores API v3, or experiments.
---

# Langfuse - OpenTelemetry-Native LLM Observability Platform

**Version:** 4.7.0 | **Last Updated:** 2026-09-26

## Overview

Langfuse is an open-source LLM observability platform:

- **Tracing**: Capture and analyze LLM calls
- **Prompt Management**: Version and manage prompts
- **A/B Testing**: Compare different prompts and models
- **Analytics**: Deep insights into LLM performance
- **Multi-Model Support**: Works with all major LLM providers

**Documentation**: https://langfuse.com/docs

## When to Use This Skill

Activate when users need:

- "Monitor LLM performance"
- "Manage and version prompts"
- "A/B test different prompts"
- "Debug LLM applications"
- "Track LLM costs and latency"

## Core Concepts

### 1. Basic Setup

```python
pip install langfuse
```

```python
from langfuse import Langfuse

# Initialize client
langfuse = Langfuse(
    public_key="pk-lf-...",
    secret_key="sk-lf-...",
    host="https://cloud.langfuse.com"  # or your self-hosted instance
)
```

### 2. Tracing LLM Calls

```python
from langfuse.decorators import observe

@observe()
def generate_response(query: str):
    """Trace a simple function."""
    return llm.generate(query)

# With explicit trace creation
trace = langfuse.trace(
    name="chat_completions",
    metadata={"user_id": "123"}
)

generation = trace.generation(
    name="gpt-4-response",
    model="gpt-4",
    input={"query": "What is AI?"},
    output={"response": "AI is..."},
    usage={"prompt_tokens": 10, "completion_tokens": 20}
)
```

### 3. Prompt Management

```python
# Create a prompt
prompt = langfuse.create_prompt(
    name="curriculum_tutor",
    prompt="You are a helpful tutor for {subject}. Help students learn {topic}.",
    config={"temperature": 0.7, "max_tokens": 500}
)

# Get prompt with variables
compiled = langfuse.get_prompt("curriculum_tutor").compile(
    subject="Mathematics",
    topic="algebra"
)

# Version prompts
prompt_v2 = langfuse.create_prompt(
    name="curriculum_tutor",
    version=2,
    prompt="You are an expert tutor in {subject}. Guide students through {topic} with examples.",
    config={"temperature": 0.5, "max_tokens": 700}
)
```

### 4. A/B Testing

```python
# Create experiment
experiment = langfuse.create_experiment(
    name="prompt_optimization",
    description="Testing different prompt styles"
)

# Run with different variants
@observe(name="variant_a")
def run_variant_a(query: str):
    return llm.generate(f"Answer this: {query}")

@observe(name="variant_b")
def run_variant_b(query: str):
    return llm.generate(f"Please provide a detailed answer to: {query}")

# Compare results
result_a = run_variant_a("What is algebra?")
result_b = run_variant_b("What is algebra?")
```

### 5. Session Tracking

```python
# Create session
session = langfuse.create_session(
    user_id="user_123",
    metadata={"subject": "Mathematics", "grade": "Junior Cycle"}
)

# Add traces to session
trace1 = session.trace(name="question_1")
trace2 = session.trace(name="question_2")

# Analyze session performance
sessions = langfuse.fetch_sessions(user_id="user_123")
```

## Advanced Features

### Evaluation Scoring

```python
from langfuse import Score

# Add manual scores
generation.score(
    name="relevance",
    value=0.9,
    comment="Highly relevant to user query"
)

# Add automated scores
generation.score(
    name="latency",
    value=generation.end_time - generation.start_time,
    comment="Response time in seconds"
)
```

### User Feedback

```python
# Capture user feedback
generation.score(
    name="user_feedback",
    value=1,  # 1-5 scale
    comment="Helpful response"
)

# Track thumbs up/down
generation.score(
    name="thumbs_up",
    value=True,
    comment="User liked the response"
)
```

### Cost Tracking

```python
# Langfuse automatically tracks costs
# Configure pricing
langfuse.configure_costs(
    model_pricing={
        "gpt-4": {"input": 0.03, "output": 0.06},
        "gpt-3.5-turbo": {"input": 0.0015, "output": 0.002}
    }
)

# Query costs
costs = langfuse.fetch_costs()
```

## Best Practices

### Tracing

1. **Granularity**: Add traces at appropriate levels (session, trace, generation)
2. **Metadata**: Include relevant metadata for filtering and analysis
3. **Context**: Capture user context and environment information

### Prompt Management

1. **Versioning**: Always version prompts when making changes
2. **Variables**: Use clear variable names in prompts
3. **Testing**: A/B test prompts before production deployment

### Analytics

1. **Regular Review**: Regularly review traces and scores
2. **Alerts**: Set up alerts for unusual patterns
3. **Optimization**: Use data to optimize prompts and models

## Configuration

### Environment Variables

```bash
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com
```

### Self-Hosting

```bash
# Docker
docker run -p 3000:3000 \
  -e LANGFUSE_SALT=your-salt \
  -e DATABASE_URL=postgresql://... \
  langfuse/langfuse

# Kubernetes
helm repo add langfuse https://langfuse.github.io/charts
helm install langfuse langfuse/langfuse
```

## Installation

```bash
pip install langfuse
# or
npm install langfuse
```

## Project Integration

### Use Cases

| Scenario | Pattern |
|----------|---------|
| Agent Monitoring | Trace all agent interactions |
| Prompt Optimization | A/B test prompt variants |
| Cost Analysis | Track LLM costs by model |
| Quality Assurance | Score and evaluate responses |

### Related Skills

- [`ragas`](.skills/ragas/SKILL.md) - RAG evaluation framework
- [`agno`](.skills/agno/SKILL.md) - Agent framework with observability
- [`google-adk`](.skills/google-adk/SKILL.md) - Google's agent framework

## Live version (verified 2026-06-29, Agent 90)

- **Python SDK v4.12.0** + **JS/TS SDK v5.9.0** on **Platform v3.125+**
- The v3 release is **OTEL-native** (no more `langfuse.trace(...).generation(...)` builder)
- New `get_client()` singleton; v4 prefers env-var auth over `Langfuse(public_key=...)` constructor
- `start_as_current_observation(as_type="generation"|"agent"|"tool"|"event")` is the v3 pattern (replaces `@observe()` with implicit type)
- Scores API v3 — typed `value` (NUMERIC/BOOLEAN/CATEGORICAL/TEXT)
- LiteLLM integration: `callbacks: ["langfuse_otel"]` + `LANGFUSE_OTEL_HOST` (was `callbacks: ["langfuse"]` + `LANGFUSE_HOST`)
- JS multi-package install: `@langfuse/tracing`, `@langfuse/otel`, `@langfuse/client`, `@langfuse/browser`, `@langfuse/openai`, `@langfuse/langchain`
- Frontend scores: `@langfuse/browser` (2026-06-18, public-key only)
- Self-host: must pin to `v3.125.0+` for Python SDK v4

## Anti-patterns (v3/v4)

1. Direct `Langfuse(public_key=..., host=...)` constructor → v4 prefers `get_client()` + env vars.
2. `openai.langfuse_public_key = "pk-lf-..."` setattr → removed in v4.
3. `callbacks: ["langfuse"]` (LiteLLM) → replaced by `callbacks: ["langfuse_otel"]`.
4. `langfuse.score(...)` with float `value` only → v3 typed value.
5. `@observe()` without explicit `as_type=` → v3 prefers context-manager.
6. `langfuse.create_experiment(...)` → replaced by `run_experiment()` + Datasets API.
7. `langfuse.fetch_sessions(...)` → replaced by `/api/public/v2/observations` + `query-via-sdk` (2026-05-15).
8. Self-host `langfuse/langfuse:latest` → must pin to **v3.125.0+** for Python SDK v4.
9. Wrapping the whole OpenAI client with `@observe()` → v3 has dedicated `from langfuse.openai import openai` drop-in.
10. Calling `langfuse.flush()` inside `@observe()` body → v3 flush is global + automatic.


## Version policy

Per [`docs/SKILL-VERSION-HEADER-POLICY.md`](../../../../docs/SKILL-VERSION-HEADER-POLICY.md) (the canonical refresh policy):

- The canonical refresh cadence is weekly (Stage 7 GH workflow) + on-demand (per openspec change)
- The canonical command is `python3 scripts/audit/refresh_skill_versions.py`
- The canonical package→skill mapping is at [`scripts/audit/PACKAGE_TO_SKILL_MAP.md`](../../../../scripts/audit/PACKAGE_TO_SKILL_MAP.md)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall