Skip to content

rbutinar/ade-ops / .agents/skills/fabric-lineage

fabric-lineage

Verified

Extract Lineage from Fabric Artifacts

/fabric-lineage — Extract Lineage from Fabric Artifacts

You are extracting lineage relationships from Fabric artifacts (views, stored procedures, pipelines) and persisting them into a metadata catalog (lineage_meta schema).

⚠️ Heavy skill — large prerequisites:

  • Needs the Fabric Warehouse SQL connector (for sys.sql_expression_dependencies native method) and/or extracted DDL from /fabric-warehouse-extract.
  • Needs the lineage metadata catalog (lineage_meta.nodes, lineage_meta.edges) — out of scope for ade-ops today.
  • Pipeline invocation lineage needs the inventory JSON produced by /fabric-extract-v2.

For today this skill is optional. The full data-lineage story will mature alongside the corresponding NotebookIOParser port (see backlog).

Prerequisites

  • One or more of:
    • Active connection to the warehouse (for native method)
    • Output of /fabric-warehouse-extract (for regex method)
    • Output of /fabric-extract-v2 (for pipeline invocation method)
  • lineage_meta.nodes + lineage_meta.edges tables (own bootstrap; out of ade-ops scope)

Usage

/fabric-lineage {target} {method} --env {env}

Where:

  • target:
    • all (default) — views + procedures + pipelines
    • views — table/view dependencies
    • procedures — SP materialization patterns (SELECT INTO, INSERT SELECT, dynamic SQL)
    • pipelines — ExecutePipeline / InvokePipeline calling other pipelines
  • method:
    • auto (default) — native first, regex fallback
    • native — sys.sql_expression_dependencies (requires warehouse connection)
    • regex — parse stored DDL (works offline against extractions)

What This Skill Extracts

View lineage

EdgeExample
Table → Viewdbo.Parts feeds dbo.v_PartDetails
View → Viewdbo.v_BaseData feeds dbo.v_Summary
Lakehouse → Viewlkh_SOP.Tables.SAP_Parts feeds dbo.v_Parts

SP materialization lineage

Detect these patterns in procedure code:

-- SELECT INTO
SELECT col1, col2 INTO dbo.mat_Summary FROM dbo.v_Source

-- INSERT SELECT
INSERT INTO dbo.mat_Target SELECT * FROM dbo.v_Source

-- Dynamic SQL (detected, NOT resolved — flagged for review)
SET @sql = 'SELECT * INTO ' + @target + ' FROM ' + @source
EXEC(@sql)

Create edges:

  • dbo.v_Source → sp_Materialize (feeds)
  • sp_Materialize → dbo.mat_Summary (produces)

Pipeline invocation lineage

Detect both formats:

// Format 1: ExecutePipeline (ADF-style)
{"typeProperties": {"pipeline": {"referenceName": "{guid}", "type": "PipelineReference"}}}

// Format 2: InvokePipeline / InvokeFabricPipeline
{"typeProperties": {"pipelineId": "{guid}", "workspaceId": "{guid}"}}

GUIDs are resolved to display names using inventory.json produced by /fabric-extract-v2.

Edges:

  • activity.Load_L3 → pipeline.Master_L3 (invokes)
  • Edge category: orchestration

Behavior

Step 1: Resolve Sources

  • Native: connect to warehouse via pyodbc + SQL token (same as /fabric-sql-deploy).
  • Regex: locate latest extraction under {project_root}/_data/extractions/.
  • Pipelines: locate latest inventory under {project_root}/_data/extractions/_inventory_*.json plus pipeline definitions.

Step 2: Build Node + Edge Sets

For each target, collect nodes (tables, views, procedures, pipelines, activities) and edges (feeds, produces, depends_on, invokes).

Each node carries:

{"key": "schema.name", "type": "view|table|procedure|pipeline|activity", "platform": "fabric", "workspace": "{ws_id}"}

Each edge:

{"from": "node_key", "to": "node_key", "type": "feeds|produces|invokes|depends_on", "confidence": "high|medium|low"}

Step 3: Persist

Upsert into lineage_meta.nodes and lineage_meta.edges. Idempotent — re-running over the same extraction must not duplicate edges.

Step 4: Report

=== FABRIC LINEAGE — {env} ===

Views:       42 nodes, 87 edges
Procedures:  6 nodes, 18 materialization edges
Pipelines:   12 nodes, 25 invocation edges

Confidence:
  high:   105 edges (literal references)
  medium: 19 edges (parameterised but resolvable)
  low:    6 edges (dynamic SQL — flagged for human review)

Persisted to lineage_meta.{nodes,edges}.

Step 5: Log

{ISO_timestamp} | {role} | FABRIC-LINEAGE | {env} | {target}/{method}: {n_nodes} nodes, {n_edges} edges | {ok|fail}

Deposit into the KB

The KB's structural feed normalizes any lineage source to one record (nodes/edges, a subset of lineage_meta) and deposits it under docs/knowledge/lineage/ (query via /knowledge lineage <object>). A Fabric adapter — normalizing the nodes/edges this skill computes into that record — is the integration point. Until it exists, this skill targets lineage_meta.{nodes,edges} directly, which (together with the ade-catalog graph) remains an external capability not yet present in this repo. See core/conventions/knowledge-base.md → Structural deposit.

Known Limitations

PatternImpactWorkaround
Dynamic SQL (EXEC(@sql))Edge marked low-confidence, target unresolvedUse sys.dm_* runtime lineage if accessible
Cross-warehouse referencesNot auto-resolved across warehousesPass both extractions in the same run
3-part vs 2-part namesNaming inconsistencyNormalise via overlay default schema

Notes

  • Native method (sys.sql_expression_dependencies) is more reliable than regex but only sees objects the user can access.
  • Regex method works offline — useful for batch processing extractions from many environments.
  • Pipeline lineage requires /fabric-extract-v2 to have run first (provides GUID → name mapping).
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall