Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/dlt-sync

dlt-sync

Verified


name: dlt-sync description: "Layer 9 of the knowledge-sync-loop — the DLT source surface validator. Use when the user asks 'is my DLT source up to date', 'how many DLT sources do we have', 'does my DLT source use the canonical helpers', 'what does sync:dlt do', 'which DLT sources have drift'. Per the 2026-08-15-dlt-sync-loop-v1 change. Triggers: 'sync:dlt', 'dlt_sources', 'dlt-source-search', '@dlt.source', '@dlt.resource', 'dlt_sources/common', 'Layer 9'."

DLT Source Sync (Layer 9 of the knowledge-sync-loop)

The Layer 9 of the 9-layer pull-based sync architecture from 2026-08-15-knowledge-sync-loop-v1 (extended by 4 follow-ups). Validates the 1903 .py files + 865 @dlt.source + 924 @dlt.resource across 13 jurisdiction subdirs at dlt_sources/.

Why Layer 9?

The 8-layer architecture from 2026-08-15-knowledge-sync-loop-v1 (extended by 2026-08-15-retroactive-pre-v7-cleanup-v1 for path drift + Layer 6 — Dagster, 2026-08-15-baml-sync-loop-v1 for Layer 7 — BAML, 2026-08-15-stacks-sync-loop-v1 for Layer 8 — IaC stacks) covered 11 of the 14 knowledge surfaces. The next-biggest remaining gap was the DLT sources surface — the 1903 .py files (865 @dlt.source + 924 @dlt.resource) in 13 jurisdiction subdirs.

DLT source drift is silent: when a developer adds a new @dlt.resource(name=...) but uses a stale write_disposition="replace" or references a destination that's no longer in the catalog, the breakage doesn't surface until the next pipeline run. The 13 subdirs each have their own patterns, and there's no canonical check that they all conform to the dlt_sources/common/ helpers.

Layer 9 closes the DLT sources gap.

What Layer 9 covers

bash scripts/sync/dlt.sh walks the 13 jurisdiction subdirs at dlt_sources/ + produces a per-subdir report to stedding/sync-reports/dlt-{date}.md with:

  • The per-subdir .py file counts
  • The @dlt.source + @dlt.resource counts
  • Drift detection (duplicate names + stale write_disposition)
  • The canonical dlt_sources/common/ helpers inventory

The orchestrator mise run sync:dlt runs all 5 sub-layers + writes a unified report to stedding/sync-reports/dlt-all-{date}.md.

The 5 sub-layers

Sub-layerTaskWhat it does
1sync:dlt-driftDetects duplicate @dlt.source / @dlt.resource names + stale write_disposition="replace"
2sync:dlt-cccAppends the 24th CCC concept guide (dlt-source-search) + reindex
3sync:dlt-cogneeIngests the 1903 DLT sources into the 13th Cognee cluster (dlt_sources)
4sync:dlt-testRuns dlt pipeline --dry-run on a sampling of the 13 subdirs
5sync:dlt-lintPer-subdir stats (GOLD_STANDARD-style report)

The new artifacts

ArtifactFilePurpose
dlt-sync skill.agents/skills/dlt-sync/SKILL.mdthis file
24th CCC guidedlt-source-searchsurfaces the 1903 DLT sources via CCC
13th Cognee clusterdlt_sourcesthe 1903 DLT source files
dlt_sync_health assetorchestration/defs/sync_assets.pyDagster asset
scripts/cognee_ingest_dlt_sources.pyingestorThe canonical Cognee cluster ingestor
notebooks/28_dlt_sync_dashboard.pydashboardLayer 9 marimo surface

DLT evolution feedback loop

The system grows its knowledge surface over time via the DLT evolution feedback loop:

dlt source file modified
  → sync:dlt-cognee detects the change
  → re-cognifies the modified source into the dlt_sources cluster
  → sync:dlt-ccc updates the 24th concept guide
  → The deployment control panel (notebook 24) surfaces the change

Quick routing

If you want to...Do this
Check the DLT source healthmise run sync:dlt
See the per-subdir breakdowncat stedding/sync-reports/dlt-$(date +%Y-%m-%d).md
Add a new DLT sourceCreate the .py file in the right subdir + use the canonical dlt_sources/common/ helpers + run sync:dlt
Fix a DLT driftsync:dlt-drift will list the broken refs; fix + re-run
See the DLT dashboardOpen notebooks/28_dlt_sync_dashboard.py
Run the DLT pipeline dry-runuv run python -c "from dlt import pipeline; pipeline()"

Cross-references

  • openspec/changes/2026-08-15-knowledge-sync-loop-v1/ (the foundation)
  • openspec/changes/2026-08-15-baml-sync-loop-v1/ (Layer 7)
  • openspec/changes/2026-08-15-stacks-sync-loop-v1/ (Layer 8)
  • dlt_sources/ (the 1903 DLT source files)
  • dlt_sources/common/ (the canonical DLT helpers)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall