Skip to content

cloudthinker-ai/CloudSkills / skills/connections/managing-apache-airflow-deep

managing-apache-airflow-deep

Unverified

Use when working with Apache Airflow Deep — apache Airflow deep management covering DAG inventory, task instance monitoring, scheduler health, executor status, pool and variable management, connection auditing, and SLA miss tracking. Use when investigating DAG failures, analyzing task duration trends, monitoring scheduler performance, or auditing Airflow configurations.

Apache Airflow Deep Management Skill

Manage and monitor Apache Airflow DAGs, task instances, scheduler, and infrastructure health.

MANDATORY: Discovery-First Pattern

Always list DAGs and check scheduler health before querying specific task instances.

Phase 1: Discovery

#!/bin/bash

AIRFLOW_API="${AIRFLOW_BASE_URL}/api/v1"

airflow_api() {
    curl -s -u "$AIRFLOW_USERNAME:$AIRFLOW_PASSWORD" \
         -H "Content-Type: application/json" \
         "${AIRFLOW_API}/${1}"
}

echo "=== Airflow Health ==="
airflow_api "health" | jq '{
    metadatabase: .metadatabase.status,
    scheduler: .scheduler.status,
    scheduler_heartbeat: .scheduler.latest_scheduler_heartbeat
}'

echo ""
echo "=== DAGs Summary ==="
airflow_api "dags?limit=50" | jq -r '
    .dags[] |
    "\(.dag_id)\t\(.is_paused)\t\(.schedule_interval // "None")\t\(.owners | join(","))"
' | column -t | head -30

echo ""
echo "=== Pools ==="
airflow_api "pools" | jq -r '
    .pools[] |
    "\(.name)\t\(.slots)\tqueued=\(.queued_slots)\trunning=\(.running_slots)\topen=\(.open_slots)"
' | column -t

Phase 2: Analysis

#!/bin/bash

echo "=== Failed DAG Runs (last 24h) ==="
YESTERDAY=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)
airflow_api "dags/~/dagRuns?start_date_gte=${YESTERDAY}&state=failed&limit=20" | jq -r '
    .dag_runs[] |
    "\(.dag_id)\t\(.dag_run_id)\t\(.state)\t\(.end_date // "running")"
' | column -t | head -20

echo ""
echo "=== Failed Task Instances (last 24h) ==="
airflow_api "dags/~/dagRuns/~/taskInstances?start_date_gte=${YESTERDAY}&state=failed&limit=20" | jq -r '
    .task_instances[] |
    "\(.dag_id)\t\(.task_id)\t\(.state)\t\(.try_number) tries\t\(.duration // 0)s"
' | column -t | head -20

echo ""
echo "=== SLA Misses ==="
airflow_api "dags/~/dagRuns/~/taskInstances?state=sla_miss&limit=10" 2>/dev/null | jq -r '
    .task_instances[]? |
    "\(.dag_id)\t\(.task_id)\t\(.execution_date)"
' | column -t

echo ""
echo "=== Variables (names only) ==="
airflow_api "variables?limit=50" | jq -r '.variables[].key' | head -20

echo ""
echo "=== Connections Summary ==="
airflow_api "connections?limit=50" | jq -r '
    .connections[] |
    "\(.connection_id)\t\(.conn_type)\t\(.host // "n/a")"
' | column -t | head -20

Output Rules

  • TOKEN EFFICIENCY: Target <=50 lines per output
  • Use date filters to scope DAG runs and task instances
  • Never dump full DAG source code or XCom values -- extract key metadata

Output Format

Present results as a structured report:

Managing Apache Airflow Deep Report
═══════════════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

  1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
  2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
  3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
  4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
  5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

ShortcutCounterWhy
"I'll skip discovery and check known resources"Always run Phase 1 discovery firstResource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"Follow the full discovery → analysis flowQuick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"Audit configuration explicitlyDefaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"Always check relevant metrics when availableAPI/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"Try the command and report the actual errorAssumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

  • Scheduler lag: If scheduler heartbeat is stale, tasks will not be scheduled -- check scheduler process
  • Pool exhaustion: All pool slots occupied means tasks queue indefinitely -- monitor open_slots
  • Zombie tasks: Tasks marked running but scheduler lost contact -- check for zombie detection in logs
  • DAG parsing errors: Import errors prevent DAGs from loading -- check import_errors endpoint
  • XCom size: Large XCom values degrade database performance -- monitor XCom table size
  • Executor capacity: Celery/Kubernetes executor has worker limits -- check worker availability
  • Database connections: Metadata DB connection pool exhaustion causes widespread failures
  • Catchup: DAGs with catchup=True can spawn hundreds of backfill runs when unpaused
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 192kUpdated Claude CodeCodexVerifiedInstall