SSZcreate/Pentab_server / .agents/skills/cybersecurity/skills/detecting-container-drift-at-runtime
detecting-container-drift-at-runtime
FlaggedDetects unauthorized runtime drift in containers by monitoring binary execution, filesystem changes, and configuration deviation from the original immutable image, using Falco and Microsoft Defender for Containers. Use when validating immutable-infrastructure controls, hunting for unexpected package installs or binaries written inside a running container, or determining whether a container diverged from the image it was built from. Keywords: drift, immutable infrastructure, new binary executed, package install, image mismatch, Falco. Do not use for detecting breakout from the container to the host - use detecting-container-escape-attempts.
Detecting Container Drift at Runtime
Overview
Container drift occurs when running containers deviate from their original image state through unauthorized file modifications, unexpected binary execution, configuration changes, or package installations. Since containers should be treated as immutable infrastructure, any drift is a potential indicator of compromise. Detection techniques leverage the DIE (Detect, Isolate, Evict) model -- an immutable workload should not change during runtime, so any observed change is potentially evidence of malicious activity.
When to Use
- When investigating security incidents that require detecting container drift at runtime
- When building detection rules or threat hunting queries for this domain
- When SOC analysts need structured procedures for this analysis type
- When validating security monitoring coverage for related attack techniques
Prerequisites
- Kubernetes cluster v1.24+ with runtime security tooling
- Falco or Sysdig for runtime drift detection
- Container image registry with image manifests available
- Familiarity with Linux filesystem layers and OverlayFS
Core Concepts
Types of Container Drift
- Binary drift: Execution of binaries not present in the original image (downloaded malware, compiled tools)
- File drift: Creation, modification, or deletion of files in the container filesystem
- Configuration drift: Changes to environment variables, mounted secrets, or runtime parameters
- Package drift: Installation of new packages via apt, yum, pip, or npm at runtime
- Network drift: New listening ports or outbound connections not expected for the workload
Detection Methods
Image-Based Comparison: Compare the running container's filesystem against its source image to identify added, modified, or removed files.
Behavioral Monitoring: Use eBPF or kernel-level monitoring to detect process execution, file access, and network activity that deviates from expected behavior.
Digest Verification: Continuously verify that running container image digests match the approved deployment manifests.
Implementation with Falco
Detecting New Binary Execution
- rule: Drift Detected (Container Image Modified Binary)
desc: Detect execution of a binary not present in the original container image
condition: >
spawned_process and
container and
not proc.pname in (container_entrypoint) and
proc.is_exe_upper_layer = true
output: >
Drift detected: new binary executed in container
(user=%user.name command=%proc.cmdline container=%container.name
image=%container.image.repository:%container.image.tag
exe_path=%proc.exepath)
priority: WARNING
tags: [container, drift]
- rule: Container Shell Spawned
desc: Detect interactive shell in a container that should be immutable
condition: >
spawned_process and
container and
proc.name in (bash, sh, dash, zsh, csh, ksh) and
not proc.pname in (container_entrypoint)
output: >
Shell spawned in container (user=%user.name shell=%proc.name
container=%container.name image=%container.image.repository)
priority: WARNING
tags: [container, drift, shell]
Detecting Package Manager Usage
- rule: Package Manager Execution in Container
desc: Detect use of package managers indicating drift
condition: >
spawned_process and
container and
proc.name in (apt, apt-get, yum, dnf, apk, pip, pip3, npm, gem, cargo)
output: >
Package manager executed in container (user=%user.name
command=%proc.cmdline container=%container.name
image=%container.image.repository)
priority: ERROR
tags: [container, drift, package-manager]
Detecting File System Modifications
- rule: Container File System Write
desc: Detect writes to container upper layer filesystem
condition: >
open_write and
container and
fd.typechar = 'f' and
not fd.name startswith /tmp and
not fd.name startswith /var/log and
not fd.name startswith /proc
output: >
File write in container (user=%user.name file=%fd.name
container=%container.name)
priority: NOTICE
tags: [container, drift, filesystem]
Implementation with Kubernetes Enforcement
Read-Only Root Filesystem
Prevent drift by making container filesystems immutable:
apiVersion: apps/v1
kind: Deployment
metadata:
name: immutable-app
spec:
template:
spec:
containers:
- name: app
image: app:v1.0@sha256:abc123...
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
volumeMounts:
- name: tmp
mountPath: /tmp
- name: cache
mountPath: /var/cache
volumes:
- name: tmp
emptyDir:
sizeLimit: 100Mi
- name: cache
emptyDir:
sizeLimit: 50Mi
Pod Security Standards Enforcement
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
Image Digest Verification
Continuous Digest Monitoring
#!/bin/bash
# Compare running container digests against approved manifest
NAMESPACE="production"
kubectl get pods -n "$NAMESPACE" -o json | jq -r '
.items[] |
.spec.containers[] |
"\(.image) \(.imageID)"
' | while read IMAGE IMAGE_ID; do
APPROVED_DIGEST=$(kubectl get deploy -n "$NAMESPACE" -o json | \
jq -r ".items[].spec.template.spec.containers[] | select(.image==\"$IMAGE\") | .image")
if [[ "$IMAGE" != *"@sha256:"* ]]; then
echo "[WARN] Container using mutable tag: $IMAGE"
fi
done
Microsoft Defender for Containers Integration
For Azure Kubernetes environments, Microsoft Defender provides built-in binary drift detection:
{
"alertType": "K8S.NODE_ImageBinaryDrift",
"severity": "Medium",
"description": "Binary executed that was not part of the original container image",
"remediationSteps": [
"Investigate the binary origin and purpose",
"Check if the container was compromised",
"Rebuild the container from a clean image",
"Enable readOnlyRootFilesystem"
]
}
Drift Response Playbook
- Detect: Alert fires on drift event (Falco, Defender, Sysdig)
- Validate: Confirm the drift is not from an approved process (init containers, config reloads)
- Isolate: Apply a deny-all NetworkPolicy to the affected pod
- Investigate: Capture container filesystem diff and process list
- Evict: Delete the drifted pod (ReplicaSet will recreate from clean image)
- Remediate: Fix the root cause (patch vulnerability, update image, tighten RBAC)
References
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 318k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 117k | Updated | Claude CodeCodex | Verified | Install |