samber/cc-skills-golang / skills/golang-observability
golang-observability
VerifiedGolang everyday observability — the always-on signals in production. Covers structured logging with slog, Prometheus metrics, OpenTelemetry distributed tracing, continuous profiling with pprof/Pyroscope, server-side RUM event tracking, alerting, and Grafana dashboards. Apply when instrumenting Go services for production monitoring, setting up metrics or alerting, adding OpenTelemetry tracing, correlating logs with traces, migrating legacy loggers (zap/logrus/zerolog) to slog, adding observability to new features, or implementing GDPR/CCPA-compliant tracking with Customer Data Platforms (CDP). Not for temporary deep-dive performance investigation (→ See `samber/cc-skills-golang@golang-benchmark` and `samber/cc-skills-golang@golang-performance` skills).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.6 KB |
| evals/evals.json | config | 43.5 KB |
| references/alerting.md | markdown | 7.9 KB |
| references/dashboards.md | markdown | 1.7 KB |
| references/logging.md | markdown | 8.4 KB |
| references/metrics.md | markdown | 23.1 KB |
| references/profiling.md | markdown | 3.8 KB |
| references/rum.md | markdown | 11.7 KB |
| references/tracing.md | markdown | 7.7 KB |
references/profiling.md
View on GitHub# Profiling and Continuous Profiling
→ See `samber/cc-skills-golang@golang-troubleshooting` skill (pprof.md) for on-demand debugging.
## What Profiling Is
Profiling analyzes the runtime behavior of your program — where CPU time is spent, how memory is allocated, which goroutines are blocked, and where lock contention occurs. While metrics tell you "the service is slow," profiling tells you "this specific function on line 42 is the bottleneck."
## On-Demand Profiling with `pprof`
pprof endpoints MUST be protected with basic auth — NEVER expose them publicly. They leak sensitive runtime information and can be abused for DoS.
→ See `samber/cc-skills-golang@golang-troubleshooting` pprof.md for the full pprof CLI reference (profile types, capturing, analyzing, commands).
The `goroutineleak` profile (`/debug/pprof/goroutineleak`, generally available since Go 1.27) is worth exposing alongside the standard profiles — it reports goroutines blocked on unreachable concurrency primitives, a signal the standard `goroutine` profile doesn't surface directly. Since Go 1.27, tracebacks for `go 1.27+` modules also carry `runtime/pprof` goroutine labels by default; if labels could leak sensitive request data into crash logs, disable with `GODEBUG=tracebacklabels=0`.
## Continuous Profiling with Pyroscope
On-demand profiling requires you to be there when the problem happens. Continuous profiling runs always-on in the background with low overhead (~2-5% CPU), so you can look at profiles after the fact. Toggle it with an environment variable.
```go
import "github.com/grafana/pyroscope-go"
func setupContinuousProfiling() {
if os.Getenv("PROFILING_ENABLED") != "true" {
return
}
_, err := pyroscope.Start(pyroscope.Config{
ApplicationName: "my-service",
ServerAddress: os.Getenv("PYROSCOPE_URL"), // e.g., http://user:pass@pyroscope:4040
ProfileTypes: []pyroscope.ProfileType{
pyroscope.ProfileCPU,
pyroscope.ProfileAllocObjects,
pyroscope.ProfileAllocSpace,
pyroscope.ProfileInuseObjects,
pyroscope.ProfileInuseSpace,
pyroscope.ProfileGoroutines,
pyroscope.ProfileMutexCount,
pyroscope.ProfileMutexDuration,
pyroscope.ProfileBlockCount,
pyroscope.ProfileBlockDuration,
},
})
if err != nil {
slog.Error("failed to start pyroscope", "error", err)
} else {
slog.Info("continuous profiling enabled", "server", os.Getenv("PYROSCOPE_URL"))
}
}
```
## Cost of Continuous Profiling
Continuous profiling adds overhead to every running instance — CPU for collecting stack samples, memory for buffering, and network for transmitting profiles to the backend. While typically low (~2-5% CPU), this cost is **per-instance and always-on**.
**Cost factors:**
- **CPU overhead** — profiling itself consumes CPU cycles. In CPU-bound services, even 2-5% overhead matters.
- **Network/storage** — profile data is continuously shipped to Pyroscope/your backend. High-replica services multiply this.
- **All profile types enabled** — each additional profile type (mutex, block, goroutine) adds incremental overhead.
**Mitigation:**
- Toggle via environment variable (`PROFILING_ENABLED`) — enable only when needed or on a subset of instances
- Start with CPU + heap profiles only; add mutex/block/goroutine profiles when investigating specific issues
- In large deployments, enable continuous profiling on a fraction of replicas (e.g., 1 in 10) rather than all of them
## When to Profile
1. Metrics show high CPU/memory usage → look at CPU/heap profiles
2. P99 latency spikes → CPU profile + mutex profile to find contention
3. Goroutine count growing → goroutine profile to find leaks
4. Before and after an optimization → compare profiles to verify improvement
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |