Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/agent-observability

agent-observability

Unverified

references/langfuse/LANGFUSE_OBSERVABILITY.md

View on GitHub
# Langfuse Observability — Tracing the Cognition Pipeline

How Langfuse provides end-to-end observability for all LLM operations in the documentation cognition pipeline, including Cognee cognify, CocoIndex flows, and Graphiti temporal updates.

## Why Langfuse for Cognition Operations

Every cognition operation involves LLM calls — entity extraction, relationship inference, summarization, and GraphRAG search. Without tracing, these calls are opaque black boxes. Langfuse provides:

- **Per-operation cost tracking**: How much did cognify cost for each dataset?
- **Latency measurement**: Which phase of cognition is slowest?
- **Quality evaluation**: Are entity extractions accurate?
- **Prompt versioning**: Which extraction prompt produced the best KG?

## Setup

### MCP Server (opencode.json)

```json
"langfuse": {
  "type": "local",
  "command": ["bunx", "-y", "@langfuse/mcp"],
  "env": {
    "LANGFUSE_PUBLIC_KEY": "infisical://dev-baile/langfuse/public_key",
    "LANGFUSE_SECRET_KEY": "infisical://dev-baile/langfuse/secret_key",
    "LANGFUSE_HOST": "https://langfuse.cianfhoghlaim.ie"
  },
  "enabled": true
}
```

### Docker Stack

```bash
cd infrastructure/stacks/langfuse
docker compose -f compose.yaml -f sidecar.yaml up -d
# Services: langfuse-web (:3000), langfuse-worker, postgres, clickhouse, redis, minio
```

## Tracing Cognee Cognify Operations

### Python Integration

```python
from langfuse import Langfuse
import httpx
import os

langfuse = Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host="https://langfuse.cianfhoghlaim.ie",
)

def cognify_with_tracing(dataset: str):
    """Run Cognee cognify with Langfuse tracing."""
    
    # Create a trace for this cognify run
    trace = langfuse.trace(
        name=f"cognee-cognify-{dataset}",
        metadata={"dataset": dataset, "pipeline": "cognition"},
    )
    
    # Span 1: Document loading
    with trace.span(name="document-count") as span:
        doc_count = get_document_count(dataset)
        span.update(metadata={"document_count": doc_count})
    
    # Span 2: Entity extraction (LLM call)
    with trace.span(name="entity-extraction") as span:
        result = httpx.post(
            "http://localhost:8100/api/v1/cognify",
            json={"datasets": [dataset]},
            timeout=600,
        )
        span.update(
            metadata={
                "dataset": dataset,
                "status": "success" if result.status_code == 200 else "failed",
                "documents_processed": doc_count,
            }
        )
    
    # Span 3: Quality evaluation
    with trace.span(name="extraction-quality") as span:
        quality = evaluate_extraction_quality(dataset)
        span.update(metadata={"entities_extracted": quality["entity_count"]})
        span.score("entity_accuracy", quality["accuracy"])
    
    trace.update(output={"status": "complete", "dataset": dataset})
    return result
```

## Monitoring Cognify Costs

### Per-Dataset Cost Tracking

```python
def track_cognify_costs():
    """Report cognition costs by dataset."""
    traces = langfuse.fetch_traces(
        name="cognee-cognify-*",
        limit=50,
    )
    
    costs = {}
    for trace in traces:
        dataset = trace.metadata.get("dataset", "unknown")
        cost = sum(span.cost for span in trace.spans)
        costs[dataset] = costs.get(dataset, 0) + cost
    
    return costs

# Example output:
# {
#   "docs-agents": 0.42,
#   "docs-bonneagar": 1.87,
#   "docs-data-eng": 2.15,
#   "docs-ml": 1.04,
#   "docs-web": 1.12,
#   "docs-context": 0.23,
#   "total": 6.83
# }
```

## Tracing CCC Indexing Operations

```python
def ccc_index_with_tracing():
    """Trace CCC indexing runs."""
    trace = langfuse.trace(name="ccc-index")
    
    with trace.span(name="codebase-scan") as span:
        result = subprocess.run(
            ["bun", "run", "ccc:index"],
            capture_output=True, text=True,
        )
        files_indexed = len(result.stdout.splitlines())
        span.update(metadata={"files_indexed": files_indexed})
    
    trace.update(output={"status": "complete"})
```

## Key Metrics to Track

| Metric | Source | Dashboard |
|:--|:--|:--|
| Cognify cost per dataset | Langfuse traces | Cost by dataset chart |
| Entity extraction quality | RAGAS evaluation | Faithfulness score |
| Cognify latency | Langfuse span duration | Latency histogram |
| Documents processed per run | Trace metadata | Throughput chart |
| LLM token usage | Span token counts | Token usage by model |
| API error rate | Trace error events | Error rate chart |

## Langfuse Web UI

Access at `https://langfuse.cianfhoghlaim.ie`:

**Trace View**: Each cognition operation rendered as a waterfall timeline:
```
┌──────────────────────────────────────────────────────────┐
│ cognee-cognify-docs-agents (2.3s, $0.42)                 │
├──────────────────────────────────────────────────────────┤
│  ├─ document-count (0.1s)         116 documents           │
│  ├─ entity-extraction (1.8s)      1,847 tokens, $0.35    │
│  ├─ relationship-inference (0.3s)  234 edges, $0.05      │
│  └─ quality-evaluation (0.1s)     accuracy: 0.94         │
└──────────────────────────────────────────────────────────┘
```

**Dashboard View**: Aggregate metrics across all cognition runs:
- Total cognition cost (daily/weekly/monthly)
- Documents processed throughput
- Average entity extraction quality
- Most expensive documents (by token count)

## Alerting via Langfuse

Configure alerts for cognition pipeline health:

```
ALERT: cognify_cost > $10/run
  → Unexpected cost spike — may indicate model fallback or error retries

ALERT: extraction_quality < 0.85
  → Entity extraction degrading — may need prompt update or model change

ALERT: cognify_latency > 300s
  → Processing timeout — may indicate overloaded LLM API or Neo4j connection issues
```
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall