Skip to content

cloudthinker-ai/CloudSkills / skills/connections/managing-dbt

managing-dbt

Verified

Use when working with Dbt — dbt (data build tool) project management and monitoring. Covers model runs, test results, source freshness, documentation generation, manifest analysis, and dbt Cloud API integration. Use when checking dbt run status, investigating test failures, analyzing model lineage, or managing dbt Cloud jobs.

dbt Management Skill

Manage and monitor dbt projects, model runs, and test results via dbt CLI and dbt Cloud API.

MANDATORY: Discovery-First Pattern

Always discover project structure and recent run status before querying specific models or tests.

Phase 1: Discovery

#!/bin/bash

# dbt Cloud API helper
dbt_cloud_api() {
    local method="${1:-GET}"
    local endpoint="$2"
    local data="${3:-}"

    if [ -n "$data" ]; then
        curl -s -X "$method" \
            -H "Authorization: Token ${DBT_CLOUD_API_TOKEN}" \
            -H "Content-Type: application/json" \
            "https://cloud.getdbt.com/api/v2/${endpoint}" \
            -d "$data"
    else
        curl -s -X "$method" \
            -H "Authorization: Token ${DBT_CLOUD_API_TOKEN}" \
            "https://cloud.getdbt.com/api/v2/${endpoint}"
    fi
}

echo "=== dbt Cloud Account Info ==="
dbt_cloud_api GET "accounts/${DBT_CLOUD_ACCOUNT_ID}/" | jq '{
    name: .data.name,
    plan: .data.plan,
    run_slots: .data.run_slots
}'

echo ""
echo "=== Projects ==="
dbt_cloud_api GET "accounts/${DBT_CLOUD_ACCOUNT_ID}/projects/" | jq -r '
    .data[] | "\(.id)\t\(.name)\t\(.connection.type // "N/A")\t\(.repository.remote_url // "N/A")"
' | column -t

echo ""
echo "=== Recent Runs ==="
dbt_cloud_api GET "accounts/${DBT_CLOUD_ACCOUNT_ID}/runs/?limit=15&order_by=-finished_at" | jq -r '
    .data[] | "\(.id)\t\(.status_humanized)\t\(.job.name // .job_id)\t\(.finished_at[0:16] // "running")"
' | column -t

Core Helper Functions

#!/bin/bash

dbt_cloud_api() {
    local method="${1:-GET}"
    local endpoint="$2"
    local data="${3:-}"

    if [ -n "$data" ]; then
        curl -s -X "$method" \
            -H "Authorization: Token ${DBT_CLOUD_API_TOKEN}" \
            -H "Content-Type: application/json" \
            "https://cloud.getdbt.com/api/v2/${endpoint}" \
            -d "$data"
    else
        curl -s -X "$method" \
            -H "Authorization: Token ${DBT_CLOUD_API_TOKEN}" \
            "https://cloud.getdbt.com/api/v2/${endpoint}"
    fi
}

# dbt Cloud Administrative API (v3)
dbt_admin_api() {
    local method="${1:-GET}"
    local endpoint="$2"
    curl -s -X "$method" \
        -H "Authorization: Token ${DBT_CLOUD_API_TOKEN}" \
        "https://cloud.getdbt.com/api/v3/accounts/${DBT_CLOUD_ACCOUNT_ID}/${endpoint}"
}

Output Rules

  • TOKEN EFFICIENCY: Target ≤50 lines per output
  • Use jq to extract only needed fields from API responses
  • Never dump full manifest.json — extract specific nodes
  • Filter runs by status at the API level

Common Operations

Job Run Dashboard

#!/bin/bash
ACCOUNT_ID="${DBT_CLOUD_ACCOUNT_ID}"

echo "=== Job Run Summary ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/?limit=50&order_by=-finished_at" | jq '
    .data | group_by(.status_humanized) | map({status: .[0].status_humanized, count: length}) |
    .[] | "\(.status): \(.count)"
' -r

echo ""
echo "=== Failed Runs (recent) ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/?limit=10&order_by=-finished_at&status=20" | jq -r '
    .data[] | "\(.id)\t\(.job.name // .job_id)\t\(.finished_at[0:16])\t\(.duration_humanized)"
' | column -t

echo ""
echo "=== Currently Running ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/?limit=10&status=3" | jq -r '
    .data[] | "\(.id)\t\(.job.name // .job_id)\t\(.created_at[0:16])\t\(.duration_humanized)"
' | column -t

Run Details and Model Results

#!/bin/bash
ACCOUNT_ID="${DBT_CLOUD_ACCOUNT_ID}"
RUN_ID="${1:?Run ID required}"

echo "=== Run Details ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/" | jq '{
    id: .data.id,
    status: .data.status_humanized,
    job_name: .data.job.name,
    duration: .data.duration_humanized,
    started: .data.started_at,
    finished: .data.finished_at,
    git_sha: .data.git_sha
}'

echo ""
echo "=== Run Artifacts (model results) ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/run_results.json" | jq -r '
    .results[] |
    select(.status == "error" or .status == "fail") |
    "\(.unique_id)\t\(.status)\t\(.execution_time | floor)s\t\(.message[0:80] // "")"
' | column -t | head -20

Test Results Analysis

#!/bin/bash
ACCOUNT_ID="${DBT_CLOUD_ACCOUNT_ID}"
RUN_ID="${1:?Run ID required}"

echo "=== Test Results Summary ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/run_results.json" | jq '
    [.results[] | select(.unique_id | startswith("test."))] |
    group_by(.status) | map({status: .[0].status, count: length}) |
    .[] | "\(.status): \(.count)"
' -r

echo ""
echo "=== Failed Tests ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/run_results.json" | jq -r '
    .results[] |
    select(.unique_id | startswith("test.")) |
    select(.status == "fail" or .status == "error") |
    "\(.unique_id | split(".") | .[-1])\t\(.status)\t\(.failures) failures\t\(.message[0:80] // "")"
' | column -t | head -15

echo ""
echo "=== Slowest Tests ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/run_results.json" | jq -r '
    [.results[] | select(.unique_id | startswith("test."))] |
    sort_by(-.execution_time) | .[:10][] |
    "\(.unique_id | split(".") | .[-1])\t\(.execution_time | floor)s\t\(.status)"
' | column -t

Source Freshness

#!/bin/bash
ACCOUNT_ID="${DBT_CLOUD_ACCOUNT_ID}"
RUN_ID="${1:?Run ID required}"

echo "=== Source Freshness Results ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/sources.json" | jq -r '
    .results[] |
    "\(.unique_id)\t\(.status)\t\(.max_loaded_at[0:16] // "unknown")\t\(.criteria.warn_after.count // "?") \(.criteria.warn_after.period // "")"
' | column -t | head -20

echo ""
echo "=== Stale Sources ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/sources.json" | jq -r '
    .results[] |
    select(.status == "warn" or .status == "error") |
    "\(.unique_id)\t\(.status)\t\(.max_loaded_at[0:16] // "unknown")\t\(.snapshotted_at[0:16])"
' | column -t

Manifest and Lineage Analysis

#!/bin/bash
ACCOUNT_ID="${DBT_CLOUD_ACCOUNT_ID}"
RUN_ID="${1:?Run ID required}"

echo "=== Model Count by Materialization ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/manifest.json" | jq -r '
    [.nodes[] | select(.resource_type == "model") | .config.materialized] |
    group_by(.) | map({type: .[0], count: length}) |
    sort_by(-.count) | .[] | "\(.type)\t\(.count)"
' | column -t

echo ""
echo "=== Models Without Tests ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/manifest.json" | jq -r '
    .nodes | to_entries |
    map(select(.value.resource_type == "model")) |
    map(select(
        [.key as $k | .value.depends_on.nodes[]? | select(startswith("test."))] | length == 0
    )) |
    .[:10][] | .value.unique_id
'

echo ""
echo "=== Models Without Descriptions ==="
dbt_cloud_api GET "accounts/${ACCOUNT_ID}/runs/${RUN_ID}/artifacts/manifest.json" | jq -r '
    [.nodes[] | select(.resource_type == "model") | select(.description == "" or .description == null)] |
    .[:10][] | .unique_id
'

Output Format

Present results as a structured report:

Managing Dbt Report
═══════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

  1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
  2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
  3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
  4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
  5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

ShortcutCounterWhy
"I'll skip discovery and check known resources"Always run Phase 1 discovery firstResource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"Follow the full discovery → analysis flowQuick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"Audit configuration explicitlyDefaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"Always check relevant metrics when availableAPI/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"Try the command and report the actual errorAssumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

  • Run status codes: 10=success, 20=error, 30=cancelled, 3=running — use status_humanized for readability
  • Artifact availability: Artifacts (manifest, run_results, sources) only available after run completes
  • API rate limits: dbt Cloud API has rate limits — avoid rapid sequential calls, use specific endpoints
  • Manifest size: manifest.json can be very large — always filter with jq, never dump entirely
  • Test naming: Tests are prefixed with test. in unique_id — schema tests auto-generate names from column/model
  • Source freshness: Only available if the job includes dbt source freshness step — not all jobs run it
  • dbt Core vs Cloud: CLI commands (dbt run, dbt test) work locally; API endpoints are dbt Cloud only
  • Job vs Run: A Job is the definition (schedule, commands), a Run is a specific execution of that Job
  • Environment branches: dbt Cloud environments may target different git branches — check environment config before comparing runs
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 318kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 318kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 318kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall