Skip to content

respanai/respan / claude-plugin/skills/respan

respan

Verified

Respan for LLM observability. Use to set up tracing or the gateway, manage prompts (create/version/deploy), run evals, experiments, and datasets, or work with traces and logs. Trigger for any Respan platform task or when instrumenting an app to send telemetry to Respan.

FileKindSize
SKILL.mdskill md2.5 KB
references/evals.mdmarkdown8.2 KB
references/gateway.mdmarkdown11.8 KB
references/monitors.mdmarkdown4.2 KB
references/prompts.mdmarkdown4.8 KB
references/tracing.mdmarkdown22.9 KB

references/monitors.md

View on GitHub
# Monitors & Automation Reference

Set up monitoring alerts, online evaluations, and automated workflows.

Docs: `https://respan.ai/docs/documentation/features/monitoring/monitors.md`

---

## Monitors

A monitor watches a metric over time and sends a notification when a threshold is crossed.

### Examples

- Error rate > 5% over 10 minutes
- Total cost > $1.00 over 5 minutes
- Average latency > 3 seconds over 15 minutes
- P95 latency > 5 seconds over 1 hour

### Setup

Monitors are configured in the platform UI: **Monitoring -> Monitors -> Create**.

A monitor has two parts:

#### 1. Trigger

Template: `When [metric] of [source] is [threshold] over [time window]`

- **Metric:** cost, latency, error rate, token usage, request count, etc.
- **Source:** all requests, specific model, specific customer, etc.
- **Threshold:** condition (greater than, less than, equals)
- **Time window:** evaluation period (1 min, 5 min, 10 min, 1 hour, etc.)

Add **Where** conditions to narrow scope:
- Filter by model, project, environment, customer, or custom metadata

#### 2. Notifications

Destinations:

| Destination | Details |
|-------------|---------|
| **Email** | Customizable subject and body |
| **Slack** | Customizable message |
| **Webhook** | Raw URL — receives JSON payload |

A single monitor can send to multiple destinations.

### Manage

- **Deploy** to make a monitor live
- **Pause/Resume** deployed monitors
- **Send test alert** to validate before deploying
- **View versions** and restore earlier versions as drafts
- **Create from chart** — select a metric on the dashboard, click to create a monitor with metric/source/interval prefilled

---

## Online Evaluations (Automation)

Automatically score production traffic in real time using deployed evaluators.

### Setup

Configure in the platform UI: **Evaluators -> Automations -> Create**.

1. **Create a condition** — filter rules for which logs to evaluate:
   - **Single log** — evaluate individual spans matching criteria
   - **Aggregation** — evaluate aggregated metrics over a window

2. **Create an automation** — connect:
   - Condition (which logs)
   - Evaluator (which scoring workflow)
   - Sampling rate (0.0-1.0, fraction of matching logs to evaluate)

3. **Check results** — view eval scores in the automation table

### Use case

> "Score 10% of production chat responses for hallucination using my LLM evaluator"

```
Condition: log_type = "chat" AND model = "gpt-4o"
Evaluator: hallucination-check
Sampling: 0.1
```

---

## Webhooks

Receive event notifications via HTTP webhooks.

### Setup

1. Go to platform: **Settings -> Webhooks -> Create**
2. Enter your endpoint URL
3. Select event types to subscribe to
4. Optionally set a webhook secret for HMAC-SHA256 verification

### Verification

Respan signs webhook payloads with HMAC-SHA256. Verify the signature:

```python
import hashlib
import hmac

def verify_webhook(payload: bytes, signature: str, secret: str) -> bool:
    expected = hmac.new(
        secret.encode(),
        payload,
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(expected, signature)
```

The signature is sent in the request headers.

---

## Metrics Dashboard

Track key indicators in the platform: **Dashboard -> Metrics**.

### Available metrics

- **Requests:** total count, requests per model/customer/API key
- **Tokens:** prompt tokens, completion tokens, total tokens
- **Cost:** per-request, per-model, per-customer
- **Latency:** average, P50, P95, P99
- **Errors:** error count, error rate
- **Time to first token (TTFT):** streaming latency

### Breakdowns

Slice metrics by:
- Model
- Customer (`customer_identifier`)
- API key
- Prompt template
- Environment
- Custom metadata keys

### Per-user analytics

Use `customer_identifier` in spans to enable:
- Per-user cost tracking and budgets
- Per-user rate limiting
- User-level quality metrics

---

## CLI Commands

```bash
respan traces list --limit 10          # List recent traces
respan traces get <trace-id>           # Get trace details
respan traces summary                  # Trace summary stats
respan logs list --limit 10            # List recent logs
respan logs get <log-id>               # Get log details
respan logs summary                    # Log summary stats
```
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall