Skip to content

zystem-io/zymtrace-skills / zymtrace/skills/troubleshoot-zymtrace-profiler

troubleshoot-zymtrace-profiler

Verified

Use when the zymtrace profiler agent is misbehaving — pods crash-looping, image pull errors, OOMKilled, CPU profiles working but no GPU profiles, NVML library not found, PC sampling not producing SASS-level data, license errors on the profiler side, or optional GPU features producing nothing (no stragglers / NCCL data, empty Models page, no GPU timeline data). Walks symptom → diagnosis → fix. Focused on the agent itself; for "no data anywhere" use troubleshoot-zymtrace-backend. Trigger phrases: "profiler not working", "profiler pods crashing", "profiler CrashLoopBackOff", "profiler ImagePullBackOff", "agent OOMKilled", "no GPU profiles", "CPU profiles work but GPU doesn't", "NVML library not found", "PC sampling not working", "profiler license rejected", "agent restart cycle", "zymtrace agent unhealthy", "fix the profiler", "no stragglers detected", "GPU Stragglers page empty", "no NCCL metrics", "Models page empty", "no Model Layer frames", "no timeline data", "No high-resolution data in this range".

Troubleshoot zymtrace Profiler

Helps the user diagnose problems with the profiler agent — the DaemonSet (or Docker / binary) that ships profiles to the backend. Backend / ClickHouse / ingest problems live in troubleshoot-zymtrace-backend. If the customer's symptom is "the UI is empty" without a clearer signal, route there instead — it covers the cross-cutting "no data" diagnostic across both sides.

Greet the user

Open warmly. Diagnostic conversations start frustrated.

👋 Sorry the profiler isn't behaving — let's track it down. Tell me what you're seeing (e.g. "agent pods CrashLoopBackOff", "CPU profiles arrive but no GPU traces", "OOMKilled every 5 min", "license rejected") and I'll walk the diagnosis with you.

Need a human?

Once the agent is healthy again, connect the zymtrace MCP to your agent (Claude Code, Codex, or Cursor) — see configure-zymtrace-mcp — to analyze GPU + CPU flamegraphs in natural language. Docs: https://docs.zymtrace.com/mcp

If the user already named a symptom, skip the prompt and jump to the matching section below.

Pre-flight

Claude runs
helm version --short && kubectl version --client
kubectl cluster-info | head -2
helm list -A | grep -iE 'profiler.*zymtrace|zymtrace.*profiler'

If the profiler release isn't installed at all, that's not a "troubleshoot" — route to install-zymtrace-profiler.

Version check (once per session). Run bash ${CLAUDE_PLUGIN_ROOT}/skills/upgrade-zymtrace-backend/scripts/check-version.sh <gateway-url> (no URL → it reads the backend Helm release). If it reports status: outdated, say so early and ask once whether to upgrade to the latest — the fix may already be in a newer release (its notes: link), and features such as straggler detection and model shapes need 26.10.3+. A profiler: line saying too far; upgrade to … means that profiler is more than 3 releases behind the backend, outside the range expected to work: offer to upgrade it to the latest before deeper debugging. Keep diagnosing either way. Details: shared/conventions.md § Check for a newer zymtrace release.

Pre-resolve what you can

Recommend defaults zymtrace / profiler. Resolve the customer's actual values first. Full policy: shared/conventions.md. Commands below use <NS> / <REL> / <PREFIX> placeholders.

VariableResolve by
Profiler release + namespacehelm list -A | grep -iE 'profiler.*zymtrace|zymtrace.*profiler'
global.namePrefix (drives DaemonSet name)helm get values <REL> -n <NS> | awk '/^\s*namePrefix:/ {print $2}' (defaults to zymtrace)
Backend release (for cross-cluster context)helm list -A | grep -iE 'backend.*zymtrace'
GPU profiling enabled?helm get values <REL> -n <NS> | grep -A1 cudaProfiler | grep enabled
Current values (so any fix re-applies cleanly)helm get values <REL> -n <NS>

Things you may need to ask:

  • The exact symptom (CrashLoopBackOff vs OOMKilled vs no-GPU-profiles are different paths).
  • Recent changes: helm upgrade, node OS update, NVIDIA driver upgrade.

Symptom router

SymptomSection
Agent pods CrashLoopBackOff / not Ready§ CrashLoopBackOff
Agent pods ImagePullBackOff§ ImagePullBackOff
Agent OOMKilled / pod restart cycle§ OOMKilled / restart cycle
CPU profiles flow, but no GPU profiles (NVIDIA or AMD)§ No GPU profiles
No off-CPU profiles§ No off-CPU profiles
No stragglers / GPU Stragglers page empty / no NCCL metrics§ No straggler or NCCL data
Models page empty or no Model Layer frames§ Models page empty or no Model Layer frames
No GPU timeline data / "No high-resolution data in this range"§ No GPU timeline data
NVML library not found in agent logs§ NVML library not found
PC sampling enabled but no SASS-level data§ PC sampling produces nothing
License rejected on the profiler side§ License rejected
UI is empty entirely (could be agent OR backend)→ troubleshoot-zymtrace-backend

If the symptom isn't listed, ask the user to describe what they're seeing and pick the closest section.

A one-shot script that gathers the most common signals:

Claude runs
bash ${CLAUDE_PLUGIN_ROOT}/skills/troubleshoot-zymtrace-profiler/scripts/diagnose-agent-not-reporting.sh <NS> <REL>

It dumps DaemonSet readiness, pod status, recent logs, license/connection signals, and a describe for any non-Running pod. Use it as the opening move for most scenarios below.


CrashLoopBackOff

Most agent CrashLoopBackOff failures are one of four causes. Walk them in order.

Claude runs
kubectl describe pod -n <NS> -l app.kubernetes.io/component=profiler | tail -60
kubectl logs -n <NS> -l app.kubernetes.io/component=profiler --tail=100 --previous 2>/dev/null

Cause A — Kernel too old for eBPF

The eBPF unwinder needs a recent enough kernel (~5.4+ is safe; 4.18 is the minimum for some features). On the failed pod's node:

NODE=$(kubectl get pod -n <NS> -l app.kubernetes.io/component=profiler -o jsonpath='{.items[0].spec.nodeName}')
kubectl get node "$NODE" -o jsonpath='{.status.nodeInfo.kernelVersion}'

If the kernel is older than 5.4, recommend a node OS upgrade. There's no in-skill workaround.

Cause B — CAP_SYS_ADMIN stripped

Pod Security Admission, OPA Gatekeeper, Kyverno, or PSP-style admission controllers can strip required capabilities. Look in describe output for:

  • pod has invalid spec: capabilities not allowed
  • Admission webhook denials mentioning SYS_ADMIN
  • seccomp denials

Fix: get the cluster admin to add an exception for the profiler namespace, or run the agent in a less-restricted namespace.

Cause C — /sys/kernel/debug not mounted on the host

Some hardened distros (Talos, Bottlerocket, some GKE Sandbox configurations) don't expose /sys/kernel/debug to privileged pods. Logs will mention failed to open /sys/kernel/debug/tracing or similar.

Fix: either remount debugfs on the host (sudo mount -t debugfs none /sys/kernel/debug — node-side, not in skill), or accept that CPU profiling won't work on those nodes and nodeSelector-exclude them.

Cause D — SELinux / AppArmor denying BPF

RHEL/CentOS with SELinux in enforcing mode can block BPF syscalls. Logs may show Operation not permitted on syscalls.

Fix: ask the cluster admin to add a SELinux policy exception, or relabel the profiler with system_u:object_r:container_runtime_exec_t:s0. This is org-specific — surface and stop.


ImagePullBackOff

Claude runs
kubectl describe pod -n <NS> -l app.kubernetes.io/component=profiler 2>/dev/null \
  | grep -A2 -iE 'failed.*pull|imagepullbackoff|errimagepull|unauthorized'

Three usual causes:

Sign in eventsCauseFix
unauthorized: authentication requiredPull secret missing or wrongkubectl get secret -n <NS> to find pull secret; set global.registry.username + password via --set on helm upgrade --install --reset-then-reuse-values --atomic
not found / manifest unknownTag doesn't existVerify at https://hub.docker.com/u/zystemio or https://github.com/orgs/zystem-io/packages; fix profiler.image.tag or services.common.imageTag
connection refused / DNS failure / private registry hostnameAir-gapped, image not mirroredMirror ghcr.io/zystem-io/zymtrace-pub-profiler:<VERSION> into the customer's registry; set global.imageRegistry. See install-zymtrace-profiler reference.md § Air-gapped install.

OOMKilled / restart cycle

Claude runs
kubectl describe pod -n <NS> -l app.kubernetes.io/component=profiler \
  | grep -A2 -iE 'oomkilled|exit code: 137|last state'
kubectl top pods -n <NS> -l app.kubernetes.io/component=profiler 2>/dev/null

If Last State: Terminated with Reason: OOMKilled (or exit code 137), the agent's memory limit is too low for the workload it's seeing. Common drivers:

  1. High GPU kernel-launch rate — every CUDA kernel costs ~25 µs of profiler CPU + heap. >10k kernels/s sustained needs more memory.
  2. PC sampling enabled on a hot workload — SASS-level data is much larger than aggregated kernel stats.
  3. --samples-per-second too aggressive — default is 19 Hz; bumping to 99 Hz quadruples CPU profile volume.

Fix: bump resource limits in the canonical values file, then re-apply.

profiler:
  resources:
    requests:
      cpu: "500m"
      memory: "512Mi"
    limits:
      cpu: "2000m"
      memory: "2Gi"

Backup the canonical file first per shared/conventions.md, then:

helm upgrade --install <REL> zymtrace/profiler \
  --namespace <NS> -f <values-file> \
  --reset-then-reuse-values --atomic --debug

If OOMKills persist after bumping to 2Gi/2 CPU, ask: is PC sampling on? Recommend turning it off in steady state and enabling on-demand for debugging only.

Full sizing reference: install-zymtrace-profiler reference.md § Resource sizing.


No GPU profiles

Agent is running, CPU profiles appear in the UI, but no GPU traces. The agent is healthy — the workload isn't loading the GPU profiler library (or isn't doing GPU work yet).

Claude runs
POD=$(kubectl get pods -n <NS> -l app.kubernetes.io/component=profiler -o name | head -1)
kubectl logs -n <NS> "$POD" --tail=2000 | grep -E 'Intercepted.*zymtrace.*implant' | head -3

If the search returns lines like:

level=info msg="Intercepted zymtrace implant at /proc/576236/root//var/lib/zymtrace/profiler/libzymtracecudaprofiler.so"

…the agent is intercepting workloads. The issue is downstream (backend ingest / ClickHouse). Hand off to troubleshoot-zymtrace-backend.

If the search is empty, no workload is being GPU-profiled. Walk the workload-side checks:

  1. cudaProfiler.enabled: true in the agent's values? helm get values <REL> -n <NS> | grep -A1 cudaProfiler. If not, enable it and helm upgrade --install ... --reset-then-reuse-values --atomic. (This one switch extracts both the CUDA and ROCm libraries.)
  2. Library on the host? kubectl debug node/<gpu-node> -it --image=busybox -- ls /host/var/lib/zymtrace/profiler (needs node-debugger RBAC). Should contain libzymtracecudaprofiler.so (NVIDIA) or libzymtracerocmprofiler.so + librocprofiler-sdk.so.1 (AMD). Absent → the agent didn't extract it; restart the DaemonSet pod on that node.
  3. NVIDIA with automatic injection? Check the rule under Settings → Agent Config → CUDA Profiler in the UI: Inject CUDA Profiler is on and its process query actually matches the workload (same container / namespace / executable names as in the query bar). If there's no rule, use manual injection (next steps) or ask the user to add one.
  4. Workload has the env var? NVIDIA: kubectl get pod <gpu-workload> -n <ws-ns> -o jsonpath='{.spec.containers[*].env[?(@.name=="CUDA_INJECTION64_PATH")]}'. AMD: the same for both ROCP_TOOL_LIBRARIES and ROCPROFILER_REGISTER_LIBRARY. Empty → patch the workload's pod spec.
  5. Workload has the volume mount? Same pod, look for mountPath: /var/lib/zymtrace/profiler or /opt/zymtrace/profiler, and check the env var points inside it. Missing → patch the workload spec.
  6. Library loaded? Add RUST_LOG=zymtracecudaprofiler=info + ZYMTRACE_CUDAPROFILER__QUIET=false (AMD: RUST_LOG=zymtracerocmprofiler=info + ZYMTRACE_ROCMPROFILER__QUIET=false) to the workload and check its logs for the profiler's startup line.
  7. Workload started before the agent? Restart the workload after the DaemonSet is Ready on its node. AMD workloads must always be (re)started with the env vars set — the ROCm profiler can't attach to a running process.
  8. Is the GPU actually busy? Kernels are profiled only while they run. An idle inference engine produces no GPU profiles — send it requests.

Workload-side recipes (vLLM, SGLang, Dynamo-Triton, TensorRT-LLM, PyTorch / training jobs, NVIDIA and AMD): install-zymtrace-profiler reference.md § Profiling real workloads.


No off-CPU profiles

Off-CPU profiling is off by default, so an empty off-CPU view usually means it was never enabled — not a broken agent. It's a host-wide setting.

  1. Enabled by flag? helm get values <REL> -n <NS> | grep -i off-cpu → look for --enable-off-cpu (or a non-zero --off-cpu-sampling-rate).
  2. Enabled by rule? Check Settings → Agent Config → Off-CPU in the UI for a rule with Off-CPU profiling on whose CEL expression matches the host (rules target hosts, not processes; an expired rule no longer applies).
  3. Neither? Ask the user before enabling it — via a rule in the UI (no redeploy) or the flag (helm upgrade ... --reset-then-reuse-values). Then wait a few minutes for data.

Docs: https://docs.zymtrace.com/features/off-cpu-profiling


No straggler or NCCL data

Straggler detection is off by default (Beta) and NVIDIA/NCCL only — RCCL/AMD is unsupported. First confirm the workload is GPU-profiled at all (§ No GPU profiles) if the user also wants the CUDA flamegraph drill-down.

  1. NCCL metrics arriving? Via MCP, run discover_metrics on the project and look for nccl. metrics (nccl.exec_time, nccl.bus_bandwidth, …). Present → collection works; an empty page is a detection question (step 5). Absent → continue.
  2. Env var on every rank? kubectl get pod <rank-pod> -n <ws-ns> -o jsonpath='{.spec.containers[*].env[?(@.name=="ZYMTRACE_CUDAPROFILER__NCCL")]}' on each pod of the job — it must be true on every process (torchrun/srun propagate it; separately launched pods don't). Missing on some ranks → those ranks report nothing.
  3. Injected via CUDA_INJECTION64_PATH but enabled only in the UI? The NCCL collection override in an Agent Config rule's GPU Profiler Config applies only to auto-injected processes; processes loading the profiler via CUDA_INJECTION64_PATH ignore it → set the env var on the workload.
  4. NCCL ≥ 2.27? kubectl exec <rank-pod> -n <ws-ns> -- python -c "import torch; print(torch.cuda.nccl.version())" — e.g. (2, 29, 2). Older → upgrade NCCL / the framework image.
  5. Metrics flow but no stragglers? Expected when: the job's communicators have < 3 ranks (2-rank and single-GPU jobs are never evaluated — nccl_stragglers returns communicators_evaluated: 0, i.e. not measured); or no rank was late in ≥ 2 consecutive 60 s windows (one-window blips aren't flagged). Only collectives count — point-to-point Send/Recv isn't covered.

Once data flows, analysis lives in optimize-gpu-workloads. Setup recipes: install reference § GPU features. Docs: https://docs.zymtrace.com/gpu/nvidia/nccl


Models page empty or no Model Layer frames

Model shapes are off by default (Beta) and need two switches.

  1. Env var on the workload? kubectl get pod <gpu-workload> -n <ws-ns> -o jsonpath='{.spec.containers[*].env[?(@.name=="ZYMTRACE_CUDAPROFILER__MODEL_SHAPE__ENABLED")]}' (AMD: ZYMTRACE_ROCMPROFILER__MODEL_SHAPE__ENABLED). Must be true, next to a working GPU profiler (§ No GPU profiles).
  2. Auto-injected workload? The Agent Config GPU Profiler Config doesn't expose model-shape settings yet — auto-injected processes still need the env var on the workload.
  3. Models page hidden? Settings → Feature Flags → Models must be on. Model Layer frames in the flamegraph don't depend on it.
  4. "No models seen in this time range" → no PyTorch model was loaded in a profiled process in range, or extraction isn't enabled for that process (step 1). "Shape not stored yet" → retry shortly. To check without the UI: bash ${CLAUDE_PLUGIN_ROOT}/skills/optimize-gpu-workloads/scripts/model-shapes.sh <gateway-url> <project-id> <start-ms> <end-ms> --query 'host.name == "<host>"' lists the models recorded for that scope and window (an empty list means none were recorded).

Expected, not bugs:

  • Duplicate model versions — the structure changed after load (vLLM weight processing, resize_token_embeddings). Cosmetic; don't raise MODEL_SHAPE__SETTLE far on CUDA-graph workloads (on vLLM, 150 s collapsed layer-frame coverage from 99.4% to 13.6%).
  • Shallow layer frames under CUDA graph replay — one layer frame directly above each kernel, nesting lost. --enforce-eager restores full depth but changes performance — short diagnostic only.
  • No layer frames on CPU flamegraphs, backward-pass kernels, or launches whose stack stops in a native runtime (e.g. CuTe DSL).
  • vLLM shows a local snapshot path as the name with HF_HUB_OFFLINE=1 → pass --served-model-name <HF id>.

Docs: https://docs.zymtrace.com/gpu/models


No GPU timeline data

The timeline needs high-resolution profiling, which is off by default and on only while a short-lived rule is active (NVIDIA only; AMD isn't supported in the UI).

  1. "No high-resolution data in this range" → high-res wasn't active for those hosts then. Check Settings → Agent Config → CUDA Profiler for a High-resolution profiling rule whose CEL matches the scope and hasn't expired, then pick a range inside it.
  2. Workload GPU-profiled? High-res applies only to processes running the zymtrace GPU profiler library (manual or auto injection) → § No GPU profiles.
  3. GPU busy during the window? Idle GPUs produce no events — send traffic while the rule is active.
  4. "Too many events to load" → the brushed window exceeds 1,000,000 events; pick a shorter window or narrow the query.

To capture a new window (with confirmation; rules must expire, ≤10 min) and analyze it: analyze-gpu-timeline. Docs: https://docs.zymtrace.com/gpu/nvidia/timeline


NVML library not found

In the agent logs:

Failed to load NVML library
Could not find libnvidia-ml.so

The profiler can't find libnvidia-ml.so to collect GPU metrics. (CUDA profiling itself doesn't depend on NVML — kernel-level profiling will still work; only the GPU metrics path is broken.)

Claude runs
POD=$(kubectl get pods -n <NS> -l app.kubernetes.io/component=profiler -o name | head -1)
kubectl exec -n <NS> "$POD" -- find / -name 'libnvidia-ml*' 2>/dev/null | head -5

If the search returns nothing, the host doesn't have the NVIDIA driver mounted into the pod — install / fix the NVIDIA device plugin or GPU operator first.

If the search returns a path, set it explicitly in the values file:

profiler:
  args:
    - "-collection-agent=..."
    - "--enable-gpu-metrics"
    - "--nvml-path=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1"   # use the path found above

Drop --nvml-auto-scan once you have the path — it's only useful for first-install discovery. Re-apply via helm upgrade --install ... --reset-then-reuse-values --atomic.


PC sampling produces nothing

The customer enabled PC sampling on a workload but the UI shows no SASS / stall data.

Claude runs
WORKLOAD_NS=<ws-ns>
WORKLOAD_POD=<workload-pod>
kubectl get pod "$WORKLOAD_POD" -n "$WORKLOAD_NS" -o jsonpath='{.spec.containers[*].securityContext.privileged}'
kubectl get pod "$WORKLOAD_POD" -n "$WORKLOAD_NS" -o jsonpath='{range .spec.containers[*].env[?(@.name=="ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING")]}{.value}{"\n"}{end}'

Two requirements that must both hold:

  1. The workload container must have securityContext.privileged: true, or be started under sudo if it's bare-metal.
  2. ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING=true must be set on the workload (not the agent).

If both look correct and PC sampling still doesn't work, check the NVIDIA driver setting on the node — ask the user to run:

# Node-side, hand to user
cat /proc/driver/nvidia/params | grep RmProfilingAdminOnly

If RmProfilingAdminOnly=1, NVIDIA's admin-only restriction is in effect. Options:


License rejected

Agent logs report:

  • license invalid
  • license expired
  • forbidden
  • unauthorized
Claude runs
POD=$(kubectl get pods -n <NS> -l app.kubernetes.io/component=profiler -o name | head -1)
kubectl logs -n <NS> "$POD" --tail=200 | grep -iE 'license|forbidden|unauthorized' | head -10

Three usual causes:

SymptomCauseFix
license expiredLicense is past expiryRenew via support@zymtrace.com or new GPU trial at https://zymtrace.com/getstarted/; update the backend values file (license lives there, not on the agent) and helm upgrade --install <backend-REL> ... --reset-then-reuse-values --atomic
forbidden / unauthorized from gatewayBackend has auth.serviceToken.enabled: true; agent missing --auth-tokenAdd profiler.args: - --auth-token=$ZYMTRACE_AUTH_TOKEN to the agent values; backup canonical file first; re-apply
license invalid immediatelyWrong license format (truncated JWT, copy-paste corruption)Re-fetch the license, update backend values, re-apply

The license is validated at the backend gateway, not on the agent — the agent only sees the result. If license fixes don't take effect, also check the backend pods (troubleshoot-zymtrace-backend § License errors).


Done

Exit when:

  • Agent pods are Running on every targeted node (kubectl get ds -n <NS> <PREFIX>-profiler shows DESIRED == READY).
  • Recent agent logs show Enabling features from license: … (the backend accepted the agent) and no connection refused / forbidden / hmac doesn't match / OOM-related errors.
  • If GPU profiling enabled: the Intercepted zymtrace implant line appears in agent logs after a representative workload starts.
  • Customer confirms profiles are visible in the UI (or, if backend-side issue is suspected, hand off to troubleshoot-zymtrace-backend).

Recap which step fixed it so future-them can pattern-match.

Security constraints

  • Never disable PC sampling, license validation, or service-token auth as a "fix" — those are symptoms, not the problem. Surface the underlying issue.
  • Never modify the canonical values file without taking a backup first per shared/conventions.md.
  • Never run kubectl delete pod / kubectl delete ds without explicit user confirmation, even if it would "force a restart".
  • Never helm upgrade without --reset-then-reuse-values --atomic.
  • Never assume node-shell / SSH access. Use kubectl exec into pods, kubectl debug node/... (with appropriate RBAC), or hand the command to the user. Same rule as the install skills.
  • Never declare "fixed" without re-running the relevant check in the Done checklist.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall