Skip to content

Altman-conquer/agentrejoin / .agents/skills/metrics-graphana

metrics-graphana

Verified

Query and manage Grafana dashboards and Prometheus metrics for Happy infrastructure. Covers grafanactl CLI usage, direct Prometheus queries through Grafana proxy, and dashboard-as-code workflows. Use when user asks about metrics, dashboards, monitoring, Grafana, Prometheus, or wants to add/modify panels.

FileKindSize
SKILL.mdskill md7.8 KB
---
name: metrics-graphana
description: >
  Query and manage Grafana dashboards and Prometheus metrics for Happy infrastructure.
  Covers grafanactl CLI usage, direct Prometheus queries through Grafana proxy,
  and dashboard-as-code workflows. Use when user asks about metrics, dashboards,
  monitoring, Grafana, Prometheus, or wants to add/modify panels.
---

# Metrics & Grafana

You are the observability operator for the Happy infrastructure. You can query live Prometheus metrics, manage Grafana dashboards as code, and investigate production behavior.

## Environment Variables

Credentials are stored in the repo root `.env` file (gitignored). Load them before running commands:

```
GRAFANA_URL=...
GRAFANA_USER=...
GRAFANA_PASSWORD=...
GRAFANA_PROMETHEUS_UID=...
```

To load in shell:
```bash
set -a; source .env; set +a
```

All commands below use `$GRAFANA_URL`, `$GRAFANA_USER`, `$GRAFANA_PASSWORD`, and `$GRAFANA_PROMETHEUS_UID` from the environment.

---

## Prerequisites

### Install grafanactl

```bash
go install github.com/grafana/grafanactl/cmd/grafanactl@latest
```

Ensure `$HOME/go/bin` is on your PATH.

### Configure grafanactl

```bash
# Load env vars first
set -a; source .env; set +a

# Create a context for the Happy Grafana instance
grafanactl config set contexts.happy.grafana.server "$GRAFANA_URL"
grafanactl config set contexts.happy.grafana.user "$GRAFANA_USER"
grafanactl config set contexts.happy.grafana.password "$GRAFANA_PASSWORD"
grafanactl config set contexts.happy.grafana.org-id 1

# Switch to the context
grafanactl config use-context happy

# Verify
grafanactl config check
```

Config file lives at `~/Library/Application Support/grafanactl/config.yaml` (macOS) or `~/.config/grafanactl/config.yaml` (Linux).

---

## grafanactl CLI Reference

### List resources

```bash
grafanactl resources list                    # List all resource types
grafanactl resources get dashboards          # List all dashboards
grafanactl resources get folders             # List all folders
```

### Pull dashboards (export to disk)

```bash
grafanactl resources pull dashboards -p ./resources -o json
grafanactl resources pull dashboards/DASHBOARD_ID -p ./resources -o json
```

### Push dashboards (deploy from disk)

```bash
# Push all dashboards from ./resources
grafanactl resources push dashboards -p ./resources

# Push a specific dashboard
grafanactl resources push dashboards/DASHBOARD_ID -p ./resources

# IMPORTANT: Use --omit-manager-fields to keep dashboards editable from the Grafana UI
grafanactl resources push dashboards -p ./resources --omit-manager-fields

# Dry run (no changes)
grafanactl resources push dashboards -p ./resources --dry-run
```

### Workflow: Edit a dashboard

```bash
# 1. Pull current state
mkdir -p /tmp/grafana-work
grafanactl resources pull dashboards -p /tmp/grafana-work -o json

# 2. Edit the JSON files (add panels, modify queries, etc.)

# 3. Push back — always use --omit-manager-fields to avoid locking the UI
grafanactl resources push dashboards -p /tmp/grafana-work --omit-manager-fields
```

> **Warning:** Pushing without `--omit-manager-fields` marks the dashboard as "provisioned"
> and locks it from UI edits. Always include this flag unless you explicitly want CLI-only management.

---

## Querying Prometheus Directly

You can query Prometheus through Grafana's datasource proxy API. This is useful for live investigation without touching the Grafana UI.

### Instant query (current value)

```bash
curl -s -u "$GRAFANA_USER:$GRAFANA_PASSWORD" \
  --data-urlencode 'query=YOUR_PROMQL_HERE' \
  "$GRAFANA_URL/api/datasources/proxy/uid/$GRAFANA_PROMETHEUS_UID/api/v1/query" \
  | python3 -m json.tool
```

### Range query (time series)

```bash
curl -s -u "$GRAFANA_USER:$GRAFANA_PASSWORD" \
  --data-urlencode 'query=YOUR_PROMQL_HERE' \
  --data-urlencode 'start=UNIX_TIMESTAMP' \
  --data-urlencode 'end=UNIX_TIMESTAMP' \
  --data-urlencode 'step=60' \
  "$GRAFANA_URL/api/datasources/proxy/uid/$GRAFANA_PROMETHEUS_UID/api/v1/query_range" \
  | python3 -m json.tool
```

### List all metric names

```bash
curl -s -u "$GRAFANA_USER:$GRAFANA_PASSWORD" \
  "$GRAFANA_URL/api/datasources/proxy/uid/$GRAFANA_PROMETHEUS_UID/api/v1/label/__name__/values" \
  | python3 -c "import json,sys; [print(n) for n in json.load(sys.stdin)['data']]"
```

### Filter metric names

```bash
# Find all RPC-related metrics
curl -s -u "$GRAFANA_USER:$GRAFANA_PASSWORD" \
  "$GRAFANA_URL/api/datasources/proxy/uid/$GRAFANA_PROMETHEUS_UID/api/v1/label/__name__/values" \
  | python3 -c "import json,sys; [print(n) for n in json.load(sys.stdin)['data'] if 'rpc' in n.lower()]"
```

---

## Key Metrics

### Application metrics (handy-server)

| Metric | Type | Description |
|--------|------|-------------|
| `rpc_calls_total` | counter | RPC calls by method and result (success, not_available, target_disconnected, timeout) |
| `rpc_call_duration_seconds_bucket` | histogram | RPC call duration by method |
| `rpc_lookup_retries_bucket` | histogram | Number of retries per socket lookup by method |
| `rpc_fetchsockets_timeouts_total` | counter | fetchSockets timeout count by context (lookup, presence) |
| `websocket_connections_total` | gauge | Active WebSocket connections by type |
| `websocket_events_total` | counter | WebSocket events by type |
| `http_requests_total` | counter | HTTP requests by method, route, status |
| `http_request_duration_seconds_bucket` | histogram | HTTP request duration by route |
| `session_cache_operations_total` | counter | Session cache hits/misses by operation |
| `session_alive_events_total` | counter | Session keepalive events |
| `machine_alive_events_total` | counter | Machine keepalive events |
| `database_records_total` | gauge | Record counts by table |
| `database_updates_skipped_total` | counter | Skipped DB updates by type |

### Useful PromQL queries

```promql
# RPC success rate by method
sum by(method) (rate(rpc_calls_total{result="success"}[5m]))
/ (sum by(method) (rate(rpc_calls_total[5m])))

# RPC failures by method and reason
sum by (method, result) (rate(rpc_calls_total{result!="success"}[5m]))

# RPC failures by type only
sum by (result) (rate(rpc_calls_total{result!="success"}[5m]))

# RPC P95 latency by method
histogram_quantile(0.95, sum by (method, le) (rate(rpc_call_duration_seconds_bucket[5m])))

# Socket lookup retry distribution (P95)
histogram_quantile(0.95, sum by (method, le) (rate(rpc_lookup_retries_bucket[5m])))

# fetchSockets timeout rate by context
sum by (context) (rate(rpc_fetchsockets_timeouts_total[5m]))

# HTTP error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Top routes by request rate
topk(10, sum by(method, route) (rate(http_requests_total[5m])))
```

---

## Dashboards

### Happy Server Application Metrics
- **ID:** `470da978-91f7-4721-be2c-cc451bf074a2`
- **Tags:** agentrejoin-server, application, websocket, http, database
- **Panels:** WebSocket connections, session cache, alive events, HTTP metrics, database stats, RPC metrics

### Adding a panel

When adding panels to a dashboard JSON, follow this pattern:
1. Use the Prometheus datasource: `{"type": "prometheus", "uid": "$GRAFANA_PROMETHEUS_UID"}`
2. Pick the next available `id` (check existing panels for max id)
3. Position with `gridPos`: `h` = height (8 standard), `w` = width (12 half, 24 full), `x` = column (0 or 12), `y` = row
4. Common panel types: `stat`, `timeseries`, `piechart`, `bargauge`, `table`
5. Set appropriate `unit`: `percentunit`, `ops`, `s`, `short`, `reqps`

---

## Tips

- Always pull before editing to get the latest state
- Use `--dry-run` on push to preview changes
- The `--omit-manager-fields` flag is essential for hybrid CLI+UI workflows
- Range queries need Unix timestamps — use `date +%s` to get current time
- When investigating metrics, start with instant queries for current state, then use range queries for trends
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall