Skip to content

oliverv/caro / .agents/skills/oma-observability

oma-observability

Unverified

Design or troubleshoot telemetry, SLOs, and incident diagnostics. Route to the relevant signal, system boundary, or vendor guide.

FileKindSize
SKILL.mdskill md14.6 KB
resources/anti-patterns.mdmarkdown45.7 KB
resources/boundaries/cross-application.mdmarkdown19.5 KB
resources/boundaries/multi-tenant.mdmarkdown17.9 KB
resources/boundaries/release.mdmarkdown12.4 KB
resources/boundaries/slo.mdmarkdown6.5 KB
resources/checklist.mdmarkdown15.0 KB
resources/examples.mdmarkdown15.7 KB
resources/execution-protocol.mdmarkdown9.5 KB
resources/incident-forensics.mdmarkdown23.3 KB
resources/intent-rules.mdmarkdown11.8 KB
resources/layers/L3-network.mdmarkdown17.6 KB
resources/layers/L4-transport.mdmarkdown14.6 KB
resources/layers/L7-application/crash-analytics.mdmarkdown17.6 KB
resources/layers/L7-application/mobile-rum.mdmarkdown15.8 KB
resources/layers/L7-application/waf.mdmarkdown16.6 KB
resources/layers/L7-application/web-rum.mdmarkdown17.3 KB
resources/layers/mesh.mdmarkdown20.0 KB
resources/matrix.mdmarkdown21.1 KB
resources/meta-observability.mdmarkdown23.5 KB
resources/observability-as-code.mdmarkdown21.8 KB
resources/signals/audit.mdmarkdown13.9 KB
resources/signals/cost.mdmarkdown17.2 KB
resources/signals/logs.mdmarkdown10.2 KB
resources/signals/metrics.mdmarkdown19.6 KB
resources/signals/privacy.mdmarkdown17.5 KB
resources/signals/profiles.mdmarkdown15.2 KB
resources/signals/traces.mdmarkdown19.1 KB
resources/standards.mdmarkdown12.7 KB
resources/transport/collector-topology.mdmarkdown11.9 KB
resources/transport/otlp-grpc-vs-http.mdmarkdown7.6 KB
resources/transport/sampling-recipes.mdmarkdown11.0 KB
resources/transport/udp-statsd-mtu.mdmarkdown6.3 KB
resources/vendor-categories.mdmarkdown17.7 KB

resources/transport/udp-statsd-mtu.md

View on GitHub
---
otel_spec: "1.x (stable API/SDK)"
---

# UDP StatsD MTU and Fragmentation Guide

## 1. Scope

This document covers StatsD metric payloads transported over UDP or Unix Domain Socket (UDS)
into an OpenTelemetry Collector `statsdreceiver` or a DogStatsD agent. It does not address
TCP or HTTP transports, or metrics emitted directly via OTLP.

Sources:
- OTel statsdreceiver: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/statsdreceiver
- DogStatsD high-throughput: docs.datadoghq.com/developers/dogstatsd/high_throughput/
- Prometheus statsd_exporter issue #35 (buffer sizing)

---

## 2. Why Fragmentation Matters for UDP

UDP is connectionless and provides no retransmission. When an IP datagram exceeds the path MTU:

- **No retransmission**: if any IP fragment is lost in transit, the kernel discards the entire
  reassembled datagram silently. StatsD has no acknowledgement mechanism, so the loss is invisible.
- **Middlebox drops**: NAT gateways, stateful firewalls, and VPN concentrators frequently drop
  IP fragments entirely, or track only the first fragment for NAT state and discard the rest.
- **Reassembly CPU overhead**: fragment reassembly at the receiver incurs kernel memory allocation
  and timer management. Under high packet rate this degrades throughput measurably.

**Rule**: keep each datagram within the path MTU so it is transmitted as a single unfragmented packet.

---

## 3. Optimal Datagram Size Table

| Network path                     | Effective MTU | Max UDP payload | Notes                                      |
|----------------------------------|---------------|-----------------|---------------------------------------------|
| External network, IPv4 Ethernet  | 1500 B        | **1472 B**      | 1500 − 20 (IP) − 8 (UDP)                   |
| External network, IPv6 Ethernet  | 1500 B        | **1452 B**      | 1500 − 40 (IPv6) − 8 (UDP)                 |
| VPN / PPPoE encapsulation        | ~1460 B       | **1432 B**      | Conservative; accounts for tunnel overhead  |
| Same-host loopback (Linux/macOS) | ~65535 B      | **~16384 B**    | Kernel loopback; practical limit ~16 K      |
| Unix Domain Socket (`unixgram`)  | N/A           | **8192 B**      | Recommended; avoids kernel socket buffer pressure |

> IPv6 minimum MTU is 1280 B (RFC 8200); on links with lower MTU, fragmentation is performed
> by the source host only; not routers; making drops more likely. Always test your actual path.

---

## 4. Verification

### 4.1 PMTUD Probe (Linux / macOS)

Find the largest datagram that reaches a destination without fragmentation:

```bash
# Linux — DF bit set, vary -s until you find the boundary
ping -M do -s 1472 -c 3 <collector-host>

# macOS equivalent
ping -D -s 1472 -c 3 <collector-host>
```

Decrease `-s` in steps of 10 until you stop seeing "Frag needed" / "Message too long" errors.
The largest passing value is your PMTU; set `max_packet_size` to that value minus 28 (IPv4)
or 48 (IPv6) to leave room for IP and UDP headers.

### 4.2 Wireshark Fragment Inspection

Capture on the collector interface and filter for IP fragments:

```
ip.flags.mf == 1
```

The **More Fragments (MF)** bit set on any packet indicates fragmentation is occurring.
A fragment offset > 0 identifies continuation fragments. If you see these in production
traffic, your client is sending oversized datagrams.

For UDS traffic, capture on a loopback or use `socat` to proxy and inspect:

```bash
socat -v UNIX-RECV:/tmp/statsd.sock UDP:127.0.0.1:8126
```

---

## 5. Client-Side Batching

Most StatsD client libraries default to sending **one metric per datagram**, which is wasteful:

- A single counter line (`my.counter:1|c`) is ~20 bytes; one UDP send per metric at 1 M metrics/min
  generates 16 K packets/sec of syscall overhead.
- Enable multi-metric batching in your client library.

| Client library     | Batching config                          | Recommended buffer   |
|--------------------|------------------------------------------|----------------------|
| dogstatsd-go       | `WithMaxMessagesPerPayload(N)`           | 1472 (external UDP)  |
| statsd (npm)       | `maxBufferSize: 1432`                    | 1432 (VPN/default)   |
| statsd-client (py) | `maxudpsize=1432`                        | 1432                 |
| Any UDS client     | Set buffer ≤ 8192                        | 8192                 |

Rules:
- For external UDP: keep buffered payload at or below the PMTU value found in section 4.1.
- For UDS (same-host): buffer up to 8192 bytes per send; larger values risk ENOBUFS under load.
- Never exceed 65507 bytes (IPv4 UDP maximum); the kernel will return EMSGSIZE.

---

## 6. OTel StatsD Receiver Configuration

Default UDP port is 8125. Switch to UDS for same-host agents to eliminate network stack overhead.

```yaml
receivers:
  statsd:
    endpoint: "0.0.0.0:8125"      # UDP; change to unixgram path for UDS
    # endpoint: "/var/run/statsd.sock"
    transport: udp                  # udp | unixgram
    aggregation_interval: 60s       # flush interval to next processor
    enable_metric_type: true        # attach metric type as attribute
    is_monotonic_counter: false     # set true for always-increasing counters
    timer_histogram_mapping:
      - statsd_type: "timing"
        observer_type: "histogram"
      - statsd_type: "histogram"
        observer_type: "histogram"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    send_batch_size: 8192
    timeout: 10s

exporters:
  otlp:
    endpoint: "otel-collector-tier2:4317"
    compression: gzip

service:
  pipelines:
    metrics:
      receivers: [statsd]
      processors: [memory_limiter, batch]
      exporters: [otlp]
```

**Bridge pattern**: `StatsD clients` → UDP/UDS → `OTel Collector statsdreceiver` → OTLP gRPC →
downstream Collector or backend. This decouples metric ingestion from processing and allows the
statsdreceiver to be scaled or replaced without changing client configuration.

For UDS on Linux, set socket permissions to allow the application user:

```bash
chmod 0660 /var/run/statsd.sock
chown root:app /var/run/statsd.sock
```

Set `SO_RCVBUF` on the collector socket to handle burst traffic (Linux default is typically 208 KB;
increase to 8–25 MB for high-throughput agents via `net.core.rmem_max`):

```bash
sysctl -w net.core.rmem_max=26214400
```
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 323kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall