Skip to content

oliverv/caro / .agents/skills/oma-observability

oma-observability

Unverified

Design or troubleshoot telemetry, SLOs, and incident diagnostics. Route to the relevant signal, system boundary, or vendor guide.

FileKindSize
SKILL.mdskill md14.6 KB
resources/anti-patterns.mdmarkdown45.7 KB
resources/boundaries/cross-application.mdmarkdown19.5 KB
resources/boundaries/multi-tenant.mdmarkdown17.9 KB
resources/boundaries/release.mdmarkdown12.4 KB
resources/boundaries/slo.mdmarkdown6.5 KB
resources/checklist.mdmarkdown15.0 KB
resources/examples.mdmarkdown15.7 KB
resources/execution-protocol.mdmarkdown9.5 KB
resources/incident-forensics.mdmarkdown23.3 KB
resources/intent-rules.mdmarkdown11.8 KB
resources/layers/L3-network.mdmarkdown17.6 KB
resources/layers/L4-transport.mdmarkdown14.6 KB
resources/layers/L7-application/crash-analytics.mdmarkdown17.6 KB
resources/layers/L7-application/mobile-rum.mdmarkdown15.8 KB
resources/layers/L7-application/waf.mdmarkdown16.6 KB
resources/layers/L7-application/web-rum.mdmarkdown17.3 KB
resources/layers/mesh.mdmarkdown20.0 KB
resources/matrix.mdmarkdown21.1 KB
resources/meta-observability.mdmarkdown23.5 KB
resources/observability-as-code.mdmarkdown21.8 KB
resources/signals/audit.mdmarkdown13.9 KB
resources/signals/cost.mdmarkdown17.2 KB
resources/signals/logs.mdmarkdown10.2 KB
resources/signals/metrics.mdmarkdown19.6 KB
resources/signals/privacy.mdmarkdown17.5 KB
resources/signals/profiles.mdmarkdown15.2 KB
resources/signals/traces.mdmarkdown19.1 KB
resources/standards.mdmarkdown12.7 KB
resources/transport/collector-topology.mdmarkdown11.9 KB
resources/transport/otlp-grpc-vs-http.mdmarkdown7.6 KB
resources/transport/sampling-recipes.mdmarkdown11.0 KB
resources/transport/udp-statsd-mtu.mdmarkdown6.3 KB
resources/vendor-categories.mdmarkdown17.7 KB

resources/layers/L4-transport.md

View on GitHub
---
otel_semconv: "1.43.0 (2026-07)"
---

# L4 Transport Layer Observability

> **DISAMBIGUATION; read before proceeding.**
>
> This file covers **observing the OSI Layer 4 (TCP/UDP/QUIC) transport in your application systems**; 
> retransmits, RTT, connection lifecycle, eBPF-based socket instrumentation, and QUIC/HTTP3 semantics.
>
> This file is **NOT** about the observability pipeline transport:
> - `../transport/udp-statsd-mtu.md`; StatsD payload sizing over UDP
> - `../transport/otlp-grpc-vs-http.md`; OTLP exporter protocol choice
> - `../transport/collector-topology.md`; OTel Collector deployment topologies
> - `../transport/sampling-recipes.md`; tail/head sampling strategies
>
> The word "transport" is overloaded. When a cross-reference says `transport/`, it refers to the
> observability pipeline. When it says "L4", "TCP/UDP/QUIC", or this filename, it refers to this file.

---

## 1. Scope

OSI Layer 4 observability for production systems: TCP, UDP, QUIC/HTTP3, eBPF auto-instrumentation,
and connection lifecycle. Out of scope: L5 session (gRPC/WebSocket → L7), L6 full TLS inspection
(use Wireshark/vendor tooling), security-focused eBPF (Cilium Tetragon/Falco → security skill).
See `../standards.md §5` for the authoritative OSI boundary decision.

---

## 2. L4 Semconv (OTel)

Source: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/network/>

| Attribute | Values | Stability | Notes |
|-----------|--------|-----------|-------|
| `network.transport` | `tcp`, `udp`, `unix`, `pipe`, `quic` | **Stable** | Set on spans and metrics describing a socket |
| `network.protocol.name` | `http`, `grpc`, `amqp`, … | **Stable** | Application protocol over the transport |
| `network.protocol.version` | `1.1`, `2`, `3` | **Stable** | `3` = HTTP/3 over QUIC |
| `network.peer.address` | IP or hostname | **Stable** | Apply IP-masking before retention; `../signals/privacy.md §IP addresses` |
| `network.connection.state` | `established`, `close_wait`, … | **Development** | Do not build SLOs against this; schema may change |
| `network.connection.type` | `wifi`, `cell`, … | **Development** | Mobile/RUM only; flag in production use |

`network.connection.*` attributes are Development tier (`../standards.md §3`). Use them in test
environments and tolerate breaking changes before promoting to production dashboards.

---

## 3. TCP Observability

### 3.1 Key Metrics

| Metric | Type | Measures | Source |
|--------|------|----------|--------|
| `tcp_retransmits_total` | Counter | Retransmit segments; rate spike = congestion or loss | hostmetrics, eBPF |
| `tcp_rtt_ms` (p50/p99) | Histogram | Round-trip time per connection | eBPF socket stats |
| `tcp_active_connections` | Gauge | In-flight established connections; growth = pool leak | `/proc/net/sockstat` |
| `tcp_time_wait_connections` | Gauge | TIME_WAIT count; high under short-lived connection churn | `/proc/net/sockstat` |
| `tcp_syn_queue_drops_total` | Counter | SYN queue overflow; SYN flood or low `tcp_max_syn_backlog` | netlink / eBPF |
| `tcp_close_wait_connections` | Gauge | CLOSE_WAIT growth = server not closing half-open connections | hostmetrics |

```promql
# Retransmit rate per node
rate(tcp_retransmits_total[5m])

# p99 RTT per service
histogram_quantile(0.99, sum by (le, service_name) (rate(tcp_rtt_ms_bucket[5m])))

# TIME_WAIT pressure alert
tcp_time_wait_connections > 10000
```

### 3.2 Collection

| Method | Mechanism | Privilege |
|--------|-----------|-----------|
| OTel hostmetrics receiver | `/proc/net/tcp`, `/proc/net/snmp`, `/proc/net/sockstat` | Root or CAP_SYS_PTRACE |
| eBPF socket filter | kprobes on `tcp_retransmit_skb`, `tcp_sendmsg` | CAP_BPF (kernel ≥ 5.8) or root |
| Linux netlink SOCK_DIAG | TCP socket state enumeration | Unprivileged in most distros |

### 3.3 Common Pitfalls

**Connection pool exhaustion.** High `tcp_active_connections` with high application latency but no TCP errors
points to pool queue saturation, not network failure. Cross-ref: `../signals/traces.md §DB connection pool patterns`.

**SYN cookie activation.** `tcp_syn_queue_drops_total` > 0 in normal traffic means `tcp_max_syn_backlog`
is too low. SYN cookies mask drop counters in `/proc/net/snmp`; use eBPF for reliable counts.

**TIME_WAIT accumulation.** Short-lived HTTP/1.1 connections exhaust ephemeral ports. Remediation:
`SO_REUSEADDR`, wider `ip_local_port_range`, or HTTP/2 multiplexing.

---

## 4. UDP Observability

| Metric | Source | Measures |
|--------|--------|----------|
| `udp_datagrams_sent_total` | `/proc/net/snmp` | Baseline send throughput |
| `udp_receive_buffer_errors_total` | `/proc/net/snmp` `RcvbufErrors` | Socket buffer overflow; increase `net.core.rmem_max` |
| `udp_no_ports_total` | `/proc/net/snmp` `NoPorts` | Drop because no listener on destination port |
| `udp_checksum_errors_total` | `/proc/net/snmp` `InCsumErrors` | Corrupted datagrams; NIC or path corruption |

OTel hostmetrics `network` scraper covers these counters automatically.
Cross-ref: `../transport/udp-statsd-mtu.md` for StatsD payload sizing constraints that affect
`udp_receive_buffer_errors_total` in high-throughput StatsD pipelines.

---

## 5. eBPF-Based L4 Observability

eBPF instruments the Linux kernel without application code changes. Three tools are in scope.

### 5.1 Tool Comparison

| Tool | CNCF Status | Primary signal | OTel output | Key use case |
|------|-------------|----------------|-------------|--------------|
| **Grafana Beyla** | CNCF Incubating (2024) | HTTP/gRPC traces + Go/C++ runtime | OTLP natively | Zero-code HTTP/gRPC auto-instrumentation via uprobes; also emits `tcp_rtt_us` histograms. Requires kernel ≥ 5.2 with BTF enabled |
| **Pixie** | CNCF Sandbox | Multi-protocol traces + L4 metrics via PxL | OTLP via `px-export` | Scriptable cluster-wide eBPF; auto-discovers pods |
| **Parca** | CNCF Sandbox | Continuous CPU + off-CPU flame graphs | pprof-compatible | Whole-cluster profiling; cross-ref `../signals/profiles.md` |

Sources: <https://grafana.com/oss/beyla/>, <https://px.dev/>, <https://parca.dev/>

**Out of scope: Cilium Tetragon.** Security-focused runtime policy enforcement (syscall filtering, process
exec tracing) belongs to a security skill. Falco is the comparable alternative for kernel security events.

### 5.2 Kernel and Privilege Requirements

| Requirement | Detail |
|-------------|--------|
| Minimum kernel | ≥ 4.14 for most eBPF features; ≥ 5.8 for CAP_BPF without root |
| Capabilities | CAP_BPF + CAP_PERFMON (kernel ≥ 5.8) or root on older kernels |
| GKE Autopilot | eBPF DaemonSets disallowed; use Standard node pools |
| EKS Fargate | No DaemonSet support; eBPF unavailable |
| AKS | Supported on standard node pools; verify with `uname -r` |

```bash
# Preflight kernel check (run in initContainer or node admission)
uname -r           # must be >= 4.14, ideally >= 5.8
capsh --print | grep cap_bpf

# Beyla deploy via Helm
helm repo add grafana https://grafana.github.io/helm-charts
helm install beyla grafana/beyla --namespace beyla --create-namespace
```

### 5.3 Pixie PxL (Conceptual)

```python
# Conceptual PxL — TCP retransmit counts per pod
import px
df = px.DataFrame(table='tcp_retransmits', start_time='-5m')
df = df.groupby(['pod', 'namespace']).agg(retransmits=('retransmits', px.sum))
px.display(df)
```

---

## 6. Connection Lifecycle Observability

```
LISTEN → SYN_RECEIVED → ESTABLISHED → (data transfer)
  → FIN_WAIT_1 → FIN_WAIT_2 → TIME_WAIT → CLOSED   (active close)
  → CLOSE_WAIT → LAST_ACK → CLOSED                  (passive close)
```

| State | Observable signal | Tooling |
|-------|------------------|---------|
| SYN → ESTABLISHED | Setup latency (SYN-ACK RTT) | eBPF `tcp_v4_connect` kprobe |
| ESTABLISHED idle | Keepalive probe success/failure | `/proc/net/tcp` + hostmetrics |
| FIN_WAIT / TIME_WAIT | TIME_WAIT gauge; ephemeral port pressure | hostmetrics `/proc/net/sockstat` |
| CLOSE_WAIT growth | Server not releasing half-open connections | hostmetrics + eBPF alert |

**TLS handshake boundary.** TLS sits at L4/L5. `tls.*` semconv is Development tier; do not build
production SLOs on it. Use eBPF uprobes on OpenSSL/BoringSSL or vendor tooling for handshake latency.
Cross-ref: `../signals/privacy.md §TLS context`.

**Keepalive tuning.** TCP keepalive interval must be shorter than the upstream LB idle timeout
(AWS ALB: 60 s; AWS NLB: 350 s; GCP CLB: 600 s default). Mismatches cause silent connection resets.

**WebSocket and gRPC streams**; full lifecycle coverage is deferred to L7 (`../standards.md §5`).
Long-lived streams mitigate TIME_WAIT accumulation but introduce reconnection-storm risk on restart.

---

## 7. QUIC / HTTP/3 Transport Semantics

QUIC (RFC 9000) is a UDP-based transport with built-in TLS 1.3, stream multiplexing, and connection
migration. HTTP/3 (RFC 9114) runs exclusively over QUIC.

### 7.1 Observability Differences from TCP

| TCP concept | QUIC equivalent | Observability impact |
|-------------|----------------|----------------------|
| 3-way SYN handshake | 1-RTT or 0-RTT | No SYN/FIN to observe; connection state is QUIC-internal |
| 5-tuple for correlation | Connection ID (opaque bytes) | Connection ID survives NAT rebinding; log it on establishment |
| `/proc/net/tcp` retransmit | QUIC loss recovery (packet number gaps) | Not visible in proc; requires QUIC-aware tooling |
| TCP FIN / RST | QUIC GOAWAY / CONNECTION_CLOSE frames | Surfaced in Envoy access logs |
| HTTP/2 HOL blocking | Per-stream flow control; no HOL blocking | Stream errors do not block other streams |

### 7.2 OTel Semconv for QUIC

```
network.transport: quic
network.protocol.name: http
network.protocol.version: "3"
```

### 7.3 0-RTT Observability

0-RTT sends data before the handshake completes. Emit a span attribute or metric label
distinguishing 0-RTT from 1-RTT connections to detect replay exposure on non-idempotent endpoints.

### 7.4 Firewall and Enterprise Deployment Considerations

Enterprise networks frequently block outbound UDP 443 and UDP 8443 because legacy firewall rules assume HTTP(S) is TCP-only. QUIC (and therefore HTTP/3) rides on UDP and silently falls back to HTTP/1.1 or HTTP/2 when UDP is blocked; the fallback is invisible to application telemetry unless you instrument it.

Detect QUIC blocking:
- Emit `network.transport` attribute on client spans; compare ratio of `quic` vs `tcp` across deployment zones
- CDN access logs (Cloudflare, Fastly) expose QUIC negotiation outcome per edge
- Alert when `quic` ratio drops below baseline (for example, office network where UDP 443 is firewalled)

Action: coordinate with network security to allowlist UDP 443/8443 for validated destinations; document the fallback behavior so regressions do not silently halve transport performance.

### 7.5 Tool Support (2026)

| Tool | QUIC support |
|------|-------------|
| Envoy 1.29+ | QUIC upstream and downstream; stats in admin API and access logs |
| Istio 1.22+ | QUIC upstream via Envoy; mTLS over QUIC experimental; see `layers/mesh.md` |
| Grafana Beyla | HTTP/3 uprobes in roadmap; verify release notes before QUIC-heavy deployments |
| Cloudflare QUIC telemetry | <https://blog.cloudflare.com/tag/quic/>; public connection migration + 0-RTT metrics |
| Google QUIC metrics | <https://research.google>; packet loss recovery timing |

---

## 8. OS-Level Integration

| Source | Signal | Cross-ref |
|--------|--------|-----------|
| systemd-journal / syslog | TCP resets, netfilter drops, NIC errors | `../signals/logs.md §OS-level log sources` |
| cgroup v2 net accounting | Per-container byte/packet counters | `../transport/collector-topology.md §kubernetes` |
| nf_conntrack | NAT and stateful firewall connection table size | Requires `nf_conntrack` kernel module |

---

## 9. Matrix Cells: L4 Row

From `../matrix.md §L4-transport`:

| Cell | Symbol | Rationale |
|------|--------|-----------|
| L4 × cross-application × traces | PARTIAL | TCP is not trace-native; log socket 5-tuple for post-hoc correlation to L7 spans only |
| L4 × * × profiles | PASS | eBPF off-CPU profiling via Parca captures kernel socket overhead and network-wait; pprof output |
| L4 × privacy (all boundaries) | PARTIAL | Source/destination IPs in TCP logs are PII (GDPR Art. 4(1)); apply masking at pipeline ingress |
| L4 × SLO and L4 × release | N/A | SLO budgets and release markers are application-layer; TCP health is infra-SLI fallback only |

---

## 10. Anti-Patterns: Candidates for `../anti-patterns.md §Section H`

**H-L4-1: Connection pool observability absent.** Pool queue saturation is invisible in TCP metrics
alone; neither retransmit rate nor error rate spikes until the pool times out. Instrument pool size,
wait time, and timeout counters at the application layer. Severity: HIGH.

**H-L4-2: QUIC adoption without HTTP/3 trace tooling validation.** Enabling QUIC without verifying
that OTel SDKs emit `network.transport: quic` and `network.protocol.version: "3"` creates a
transport-layer blind spot. Add a canary assertion in staging before production rollout. Severity: HIGH.

**H-L4-3: eBPF agent deployed without kernel/capability preflight.** Beyla or Pixie DaemonSets on
incompatible kernels fail silently with no operator-visible error. Add an `initContainer` asserting
kernel version and CAP_BPF presence; fail non-zero if requirements are unmet. Severity: HIGH.

**H-L4-4: WebSocket reconnection storm undetected.** Server restart without graceful WebSocket drain
causes simultaneous mass-reconnect. TCP SYN counts look like normal new connections. Instrument client
reconnection counters and alert on rate spikes; implement exponential backoff + jitter. Severity: MEDIUM.

---

## 11. References

- OTel network attributes registry: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/network/>
- Grafana Beyla: <https://grafana.com/oss/beyla/>
- Pixie (CNCF Sandbox): <https://px.dev/>
- Parca (CNCF Sandbox): <https://parca.dev/>
- QUIC RFC 9000: <https://www.rfc-editor.org/rfc/rfc9000>
- HTTP/3 RFC 9114: <https://www.rfc-editor.org/rfc/rfc9114>
- Cloudflare QUIC telemetry: <https://blog.cloudflare.com/tag/quic/>
- OTel hostmetrics receiver: <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver>
- Linux `/proc/net/tcp` format: <https://www.kernel.org/doc/html/latest/networking/proc_net_tcp.html>
- Semconv stability tiers: `../standards.md §3`
- OSI boundary decisions: `../standards.md §5`
- Coverage matrix (L4 row): `../matrix.md §L4-transport`
- IP masking rules: `../signals/privacy.md §IP addresses`
- Profiling (eBPF/Parca): `../signals/profiles.md`
- OS-level logs: `../signals/logs.md §OS-level log sources`
- Collector Kubernetes topology: `../transport/collector-topology.md`
- StatsD UDP sizing: `../transport/udp-statsd-mtu.md`
- Service mesh + QUIC: `layers/mesh.md`
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall