monte-carlo-data/mc-agent-toolkit / skills/push-ingestion
push-ingestion
VerifiedExpert guide for Monte Carlo's push ingestion model. Use this skill whenever a customer or engineer mentions: pushing data to Monte Carlo, the IngestionService, pycarlo push APIs, build me a collection script, push metadata/lineage/query logs, invocation_id tracing, custom lineage nodes or edges, deleting push tables, or any question about why pushed data is not showing up. Also trigger when they ask to generate code that collects metadata, table schema, row counts, freshness, lineage, or query history from any data warehouse or data source and sends it to Monte Carlo. If the user mentions any warehouse, database, or data platform alongside any Monte Carlo topic, this skill is almost certainly relevant.
references/push-lineage.md
View on GitHub# Pushing Table and Column Lineage
## Overview
Both table-level and column-level lineage use the same endpoint: `POST /ingest/v1/lineage`.
The `event_type` field distinguishes them:
- `LINEAGE` — table-level: source table → destination table
- `COLUMN_LINEAGE` — column-level: source table.column → destination table.column
(also automatically creates the parent table-level edge)
Push lineage is **typically visible in the MC lineage graph within seconds to a few minutes**
via the fast direct path (PushLineageProcessor → S3 CSVs → neo4jLineageLoaderPrivate → Neo4j).
**Expiration**:
- Pushed **table lineage does not expire** (`expire_at = 9999-12-31`).
- Pushed **column lineage expires after 10 days** (same as pulled column lineage).
**Batching**: For large numbers of lineage events, split into batches. The compressed request
body must not exceed **1MB** (Kinesis limit).
## pycarlo models
```python
from pycarlo.features.ingestion import (
IngestionService,
LineageEvent,
LineageAssetRef,
ColumnLineageField,
ColumnLineageSourceField,
)
```
## Table lineage example
```python
event = LineageEvent(
destination=LineageAssetRef(
database="analytics",
schema="public",
table="customer_orders",
),
sources=[
LineageAssetRef(database="analytics", schema="public", table="customers"),
LineageAssetRef(database="analytics", schema="public", table="orders"),
],
)
result = service.send_lineage(
resource_uuid="<your-resource-uuid>",
resource_type="data-lake",
events=[event],
)
invocation_id = service.extract_invocation_id(result)
print("invocation_id:", invocation_id)
```
## Column lineage example
```python
event = LineageEvent(
destination=LineageAssetRef(
database="analytics",
schema="public",
table="customer_orders",
),
sources=[
LineageAssetRef(database="analytics", schema="public", table="customers"),
LineageAssetRef(database="analytics", schema="public", table="orders"),
],
# column mappings: dest_col ← src_table.src_col
fields=[
ColumnLineageField(
destination_field="customer_id",
source_fields=[
ColumnLineageSourceField(
database="analytics", schema="public",
table="customers", field="customer_id",
)
],
),
ColumnLineageField(
destination_field="order_amount",
source_fields=[
ColumnLineageSourceField(
database="analytics", schema="public",
table="orders", field="amount",
)
],
),
],
)
result = service.send_lineage(
resource_uuid=resource_uuid,
resource_type="data-lake",
events=[event],
)
```
Column lineage push automatically creates a table-level edge too, so you don't need to
send separate table and column lineage events for the same relationship.
## Extracting lineage from SQL logs
For warehouses that don't expose a native lineage table, extract lineage by parsing query
history SQL for `CREATE TABLE AS SELECT`, `INSERT INTO ... SELECT`, and `MERGE INTO` patterns.
Simplified example regex:
```python
import re
CTAS_PATTERN = re.compile(
r"CREATE\s+(?:OR\s+REPLACE\s+)?TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?(\S+)\s+AS\s+SELECT",
re.IGNORECASE,
)
INSERT_PATTERN = re.compile(
r"INSERT\s+(?:OVERWRITE\s+)?(?:INTO\s+)?(\S+).*?FROM\s+(\S+)",
re.IGNORECASE | re.DOTALL,
)
```
For Snowflake, BigQuery, and Redshift the query history tables provide this SQL.
For Databricks, use `system.access.table_lineage` directly (no parsing needed).
For Hive, parse the HiveServer2 log file.
## Output manifest (include invocation_id)
```python
manifest = {
"resource_uuid": resource_uuid,
"invocation_id": service.extract_invocation_id(result), # ← save this
"collected_at": datetime.now(tz=timezone.utc).isoformat(),
"edges": [
{
"destination": {"database": e.destination.database, "table": e.destination.table},
"sources": [{"database": s.database, "table": s.table} for s in e.sources],
}
for e in events
],
}
with open("lineage_output.json", "w") as f:
json.dump(manifest, f, indent=2)
```
## How push lineage is distinguished from query-derived lineage
Push-ingested lineage nodes and edges carry `origin = push_ingest` in Neo4j and
`origin_type = DIRECT_LINEAGE` in the normalized lineage model. This prevents the lineage
DAG from overwriting them with query-log-derived edges and gives MC a clear audit trail.
## Neo4j node expiry
Push-ingested **table lineage** nodes and edges are written with `expire_at = 9999-12-31`
(never expire). This is handled internally by PushLineageProcessor — you do not need to set
this manually when using `send_lineage()`.
Push-ingested **column lineage** expires after **10 days**, same as pulled column lineage.
For custom nodes created via GraphQL mutations, you **do** need to set
`expireAt: "9999-12-31"` explicitly — see `references/custom-lineage.md`.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |