ccoalm/ccl-skills / skills/platform-observability
platform-observability
VerifiedUse when designing, reviewing, debugging, or shipping observability — service logs, metrics, distributed tracing, log/trace correlation, 给接口·服务加结构化日志与 trace 透传, dashboards, alerts, on-call routing(值班/排班 SOP、P0/P1 打断;发布值班/回滚除外), SLI/SLO, error budgets, 线上是否已启用(开关·配置在生产的实际生效状态)— for a backend product. Owns the cross-cutting evidence layer — what signals must exist, what fields must propagate, what middleware must auto-wire, what verification proves a change is observable in production. Hand off mesh/routing/mTLS to `platform-service-connectivity`, release gates and rollback evidence to `platform-release-engineering`, service-internal architecture (HTTP/RPC/DB/queue) to `software-architecture`. Product-agnostic.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 35.8 KB |
| agents/openai.yaml | config | 282 B |
| references/alerting-and-on-call.md | markdown | 4.4 KB |
| references/framework-middleware-checklist.md | markdown | 18.9 KB |
| references/infra-component-deployment.md | markdown | 13.6 KB |
| references/log-correlation-recipe.md | markdown | 9.8 KB |
| references/log-schema-canonical.md | markdown | 8.0 KB |
| references/metrics-conventions.md | markdown | 12.4 KB |
| references/obs-stack-architecture.md | markdown | 9.4 KB |
| references/sli-slo-design.md | markdown | 6.4 KB |
| references/source-register.md | markdown | 1.7 KB |
references/log-schema-canonical.md
View on GitHub# Canonical Log Schema (Concrete Field Set)
The authoritative field list for structured platform logs, derived from a working production deployment. Use as a template; localize header names but preserve the field shape.
## Required fields (every log line)
| Field | Type | Purpose |
|---|---|---|
| `_ts` | keyword | ISO-8601 timestamp with timezone (e.g. `2026-05-20T14:23:01.123+08:00`) |
| `_datetime` | date | Parsed timestamp for the search backend |
| `_timestamp` | long | Unix ms (Asia/Shanghai or UTC; document the choice) |
| `_level` | keyword | TRACE / DEBUG / INFO / WARN / ERROR / FATAL (OTel-spec severity text) |
| `_msg` | text | Message body |
| `_logid` | keyword | Platform log-id (correlation handle, generated at edge if absent) |
| `_trace_id` | keyword | OTel trace id (hex) |
| `_span_id` | keyword | OTel span id (hex) |
| `_trace_flags` | keyword | OTel trace flags |
| `_psm` | keyword | Service identity (PSM-style or `service.name` OTel attr) |
| `_lane` / `_env` | text | Lane / environment label |
| `_idc` | text | Region / data center |
| `_cluster` | text | Kubernetes cluster name |
| `_pod` | text | Pod name |
| `_addr` | ip | Service IP (pod IP) |
| `_caller` | text | Source file:line where the log was emitted (file.go:123 form) |
| `_language` | text | Programming language (go, python, java, ...) |
Fields prefixed `_` are platform-reserved. Application fields use `app_*` or `biz_*` prefix to prevent collision.
`_stack` (type `text`) is set only for ERROR and above; contains the full stack trace.
## Why each field
- `_logid` — human-friendly correlation id; the only id a user reports in a ticket.
- `_trace_id` + `_span_id` — cross-service trace join; OTel-managed.
- `_caller` — source location; powerful for diff-based canary checks (see `platform-release-engineering/references/lane-orchestration-control-plane.md` § canary algorithm). Two services that emit "DB timeout" from different code lines are correctly distinguished.
- `_language` — multi-language platforms need this to group / filter log queries; also useful for stack-format detection.
- `_addr` / `_pod` — drill from "lane has problem" → "this specific pod" → "this specific replica".
- `_idc` / `_cluster` — multi-region / multi-cluster drilldown.
## Index template (Elasticsearch / OpenSearch)
```yaml
setup.template:
enabled: true
overwrite: true
pattern: "bizlogs-*" # localize prefix
fields: fields.yml
order: 140
settings:
index.number_of_shards: 3
_source.enabled: true
mappings:
dynamic: true
dynamic_date_formats: "strict_date_optional_time"
date_detection: true
```
3 shards is a reasonable default for cluster sizing 10-50 nodes. Adjust based on ingest rate.
## ILM policy
Hot 7 days, then delete is a common defensible default for biz logs:
```yaml
policy:
phases:
hot:
min_age: "0ms"
actions:
rollover:
max_age: "7d"
max_size: "30gb"
delete:
min_age: "7d"
actions:
delete: {}
```
Adjust by:
- **Compliance** — some industries require 90 days+ of audit logs (set a separate index/policy for audit-class logs).
- **Cost** — warm tier (cheaper storage) for 30-90 days is a middle ground.
- **Investigation cadence** — if incidents are typically detected and root-caused within 7d, 7d hot is sufficient.
Separate ILM policies for `audit-*` (long retention), `bizlogs-*` (7d), `debug-*` (1-3d) prevents pinning all retention to the strictest requirement.
## Filebeat configuration shape
```yaml
filebeat.inputs:
- type: log
id: bizlogs
enabled: true
paths:
- /opt/<org>/app/log/*.log
- /opt/<org>/app/log/*/*.log
- /opt/<org>/app/log/*/*/*.log # per-service/per-lane subdirs
fields_under_root: true
json:
keys_under_root: true
overwrite_keys: false # CRITICAL: see "Platform-field protection" below
add_error_key: true
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- script:
lang: javascript
id: extract_ts
file: ${path.config}/extract_ts.js # custom timestamp parser if needed
- timestamp:
field: "_timestamp"
target_field: "_datetime"
layouts: ["UNIX_MS"]
timezone: "Asia/Shanghai" # or UTC
```
`fields_under_root: true` + `json.keys_under_root: true` together mean: the JSON log line's fields become root-level fields in the indexed document. So `_logid` from the log → `_logid` in Elasticsearch. No nested `parsed.<field>` digging.
### Platform-field protection
**Do NOT set `overwrite_keys: true` with `keys_under_root: true` for reserved fields.** A compromised or malicious service could emit `{"_lane":"prod", "_level":"INFO", "_msg":"..."}` from a non-prod pod, and the log pipeline would index it as if it came from prod with INFO severity — corrupting canary queries, audit searches, and incident triage.
Enforcement options (pick at least one):
1. **Logstash filter strips/renames `_`-prefixed fields supplied by the app**, then re-injects authoritative `_lane`, `_psm`, `_idc`, `_cluster`, `_pod`, `_addr` from metadata the pipeline knows for sure (e.g. from k8s pod annotations the filebeat shipper adds).
2. **Filebeat `processors.drop_fields`** removes a denylist of app-supplied platform fields before they hit logstash.
3. **Application logger contract**: SDK is the only code path that writes `_`-prefixed fields; lint rule rejects app code that touches them directly.
Layered defense is best: SDK contract (#3) for the common case + pipeline strip (#1) as the durable guardrail in case the SDK is bypassed.
## Log path layout
Services write structured JSON logs to disk (not stdout in this pattern). Layout:
```
/opt/<org>/app/log/
<psm-1>/
<lane-A>/
app.log
app-error.log
<lane-B>/
...
<psm-2>/
...
```
Why files not stdout (in this pattern):
- Per-service per-lane separation: cleaner debugging.
- File rotation (size + time) is well-tooled.
- Filebeat reads efficiently with offset tracking.
- App can keep writing during transient filebeat outages (file buffer absorbs).
Why stdout in other patterns (mature alternative):
- k8s-native; logs visible to `kubectl logs`.
- No volume mount needed.
- Cloud-native log shippers (Cloud Logging, etc.) usually pull from stdout.
Pick one per platform; do not mix.
## In-pod sidecar vs DaemonSet collection
Two patterns:
**DaemonSet** (one filebeat per node, hostPath mount):
```
Node
├─ filebeat (DaemonSet pod)
│ └─ reads /opt/<org>/app/log/* via hostPath
└─ App pods write to /opt/<org>/app/log/<psm>/<lane>/
```
Pros: one filebeat process per node, low overhead.
Cons: hostPath access required; not compatible with serverless k8s.
**Sidecar** (one filebeat per app pod):
```
App Pod
├─ app container writes to emptyDir
└─ filebeat container reads emptyDir
```
Pros: works in serverless k8s (no hostPath); per-pod resource accounting.
Cons: filebeat container per pod multiplies resource cost.
Many platforms use **both**, selected by pod environment: serverless k8s (e.g. cloud-provider container instance) → sidecar; node-attached k8s → DaemonSet.
## Output endpoints by environment
Online (production) and offline (staging/dev) typically have different log backends:
```yaml
output:
elasticsearch:
hosts: [{{ online_es_host or offline_es_host }}]
setup.kibana:
host: {{ online_kibana_host or offline_kibana_host }}
```
Reasons to split:
- Cost: offline ES can be smaller, single-node, no replicas.
- Isolation: a dev mistake bursting logs doesn't pressure prod ES.
- Compliance: prod logs may need stricter access control.
## Verification
- Open a service log line in kibana → all required fields present, no missing fields.
- Search `_trace_id:<id>` → returns all hops of one trace.
- Search `_caller:<source-location>` → returns the same error pattern across the fleet.
- ILM rollover triggers at 7d / 30gb (test by writing high volume to a non-critical index).
- Cluster failure of online ES → filebeat buffers locally; resumes when ES returns.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |