rbutinar/ade-ops / .agents/skills/fabric-extract-v2
fabric-extract-v2
VerifiedParallel Fabric Workspace Metadata Extraction
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.9 KB |
SKILL.md
View on GitHub---
name: fabric-extract-v2
description: Parallel Fabric Workspace Metadata Extraction
---
# /fabric-extract-v2 — Parallel Fabric Workspace Metadata Extraction
You are extracting **metadata** (workspaces, semantic models, notebooks, pipelines, report visuals) from one or more Microsoft Fabric workspaces into a structured catalog.
> ⚠️ **Heavy skill**: this is a port of the ADE lab `fabric-extract-v2` workflow. It depends on the `ade_app.platforms.fabric.*` modules and on a SQL Server backend (`fabric_infra` + `powerbi_meta` schemas). **Not yet self-contained** inside ade-ops — see backlog tasks for porting `msal_cache` (auth), `TokenManager`/`CheckpointManager` (extraction infra), and the metadata schema.
>
> For <client> <project> today this skill is **optional** — it's used to populate a corporate-wide Fabric catalog, not for <project>-specific operations. Keep it documented so the porting path is clear.
## Prerequisites
- Python: `msal`, `httpx`, `pyodbc`
- `core.platforms.fabric.auth.get_fabric_token` (or the framework's `FabricConnector` once `msal_cache` is ported)
- SQL Server with `fabric_infra` and `powerbi_meta` schemas (DB+schema bootstrap is out of scope for ade-ops today — backlog item)
- Contributor access on the workspaces being extracted
## Usage
```
/fabric-extract-v2 [phase] [--workspace-filter {substring}]
```
Phases:
- `inventory` (~1-2 min for 75 workspaces) — workspace + item catalog via REST (no item content)
- `content` — parallel extraction of SemanticModel TMDL, Notebook source, Pipeline + activity definitions, Report (PBIR) pages/visuals
- `enrich` — post-extraction lineage (M partitions → data sources, DAX → dependencies); operates on already-extracted TMDL files
- `both` (default) — inventory + content
- `all` — inventory + content + enrich
## Behavior
### Phase 1 — Inventory
For each workspace matching `--workspace-filter`:
1. `mcp__databricks__...` is **not** used here — we go through `FabricConnector.client.list_workspaces()` / `list_items()` directly.
2. Persist `workspaces`, item counts by type, last-modified to `fabric_infra.workspaces` + `fabric_infra.items`.
3. Snapshot to `_data/extractions/_inventory_{timestamp}.json` for offline review.
### Phase 2 — Content (parallel)
For each item discovered in Phase 1, call `connector.client.get_item_definition(workspace_id, item_id)` to get the inline-base64 parts, then:
- **SemanticModel** — decode TMDL parts → parse tables, columns, measures, relationships; persist to `powerbi_meta.{datasets,tables,columns,measures,relationships}`.
- **Notebook** — decode `notebook-content.py` (Spark) or `notebook.ipynb`; store source code.
- **Pipeline** — decode `pipeline-content.json`; parse activities into `fabric_infra.pipelines` + `pipeline_activities`.
- **Report** — decode PBIR parts (`pages.json`, visuals); persist `powerbi_meta.{reports,report_pages,visuals}` with measure bindings.
**LRO pattern** (critical):
```
POST .../getDefinition → 202 + Location header
poll Location until status == "Succeeded"
GET /v1/operations/{operationId}/result → actual definition
```
Do **not** rely on `resourceLocation` — it's null for `getDefinition`.
Use a `CheckpointManager` for resume-after-interruption. Force-refresh tokens on 401 (the `FabricConnector` already does this; if calling REST directly, replicate the pattern from `core/connectors/fabric.py`).
### Phase 3 — Enrich
Operate on already-extracted TMDL/PBIR files:
- **M partition parsing** — extract source connections (`lakehouse → table`, `warehouse → table`).
- **DAX dependency parsing** — extract referenced tables/measures from DAX expressions.
- Populate `powerbi_meta.data_sources`, `powerbi_meta.depends_on_tables`, `powerbi_meta.depends_on_measures`.
### Output
Imports directly to SQL Server (default backend):
- `fabric_infra.workspaces / items / lakehouses / warehouses / pipelines / notebooks / pipeline_activities`
- `powerbi_meta.reports / datasets / tables / columns / measures / relationships / report_pages / visuals / data_sources`
Snapshots to `_data/extractions/` for offline review (gitignored).
## Logging
```
{ISO_timestamp} | {role} | FABRIC-EXTRACT | -- | phase={phase} workspaces={n}: items={n} | {ok|fail}
```
## Port Status / Backlog
- `msal_cache` (token cache) → planned in `core/platforms/fabric/auth/`
- `TokenManager` / `CheckpointManager` → planned in `core/platforms/fabric/extract/`
- Metadata catalog schema (SQL Server) → out of scope for ade-ops today; users must bootstrap their own
## Notes
- Token scope: `analysis.windows.net/powerbi/api/.default` (PBI Service) works on ALL workspaces; `https://api.fabric.microsoft.com` (Fabric API) works only on Premium/Fabric capacity.
- Windows: use `\\?\` prefix for paths > 260 chars (common with long workspace + TMDL filenames).
- Excel export: `python tasks/generate_pbi_excel.py` produces 6 sheets (Summary, Reports, Measures, Tables, Relationships, Visuals).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |