monte-carlo-data/mc-agent-toolkit / skills/push-ingestion
push-ingestion
VerifiedExpert guide for Monte Carlo's push ingestion model. Use this skill whenever a customer or engineer mentions: pushing data to Monte Carlo, the IngestionService, pycarlo push APIs, build me a collection script, push metadata/lineage/query logs, invocation_id tracing, custom lineage nodes or edges, deleting push tables, or any question about why pushed data is not showing up. Also trigger when they ask to generate code that collects metadata, table schema, row counts, freshness, lineage, or query history from any data warehouse or data source and sends it to Monte Carlo. If the user mentions any warehouse, database, or data platform alongside any Monte Carlo topic, this skill is almost certainly relevant.
references/custom-lineage.md
View on GitHub# Custom Lineage Nodes and Edges
## When to use this
The `send_lineage()` pycarlo method is the right choice for warehouse tables you own.
The **GraphQL mutations** in this document are for:
- Non-warehouse assets: dbt models, Airflow DAGs, Fivetran connectors, custom ETL jobs
- Connecting nodes across different MC resources (warehouses)
- One-off lineage corrections not tied to a collector run
- Fine-grained control over node properties, object types, and expiry
All mutations use the **GraphQL API key** (not the Ingestion key) and the endpoint
`https://api.getmontecarlo.com/graphql`.
## Critical: expireAt
If you don't set `expireAt`, nodes and edges expire after **7 days** and vanish from the
lineage graph silently. For any node or edge that should persist:
```
expireAt: "9999-12-31"
```
This is the same value that `PushLineageProcessor` uses internally for all push-ingested
lineage. Forgetting this is the most common cause of "my lineage disappeared after a week".
---
## createOrUpdateLineageNode
Creates or updates a node in the lineage graph. If a node with the same
`objectType` + `objectId` + `resourceId` already exists, it is updated.
```graphql
mutation CreateOrUpdateLineageNode(
$objectType: String!
$objectId: String!
$resourceId: UUID
$resourceName: String
$name: String
$properties: [ObjectPropertyInput]
$expireAt: DateTime
) {
createOrUpdateLineageNode(
objectType: $objectType
objectId: $objectId
resourceId: $resourceId
resourceName: $resourceName
name: $name
properties: $properties
expireAt: $expireAt
) {
node {
mcon
displayName
objectType
isCustom
expireAt
}
}
}
```
**Variables:**
```json
{
"objectType": "table",
"objectId": "analytics:analytics.orders",
"resourceId": "<warehouse-uuid>",
"name": "orders",
"expireAt": "9999-12-31"
}
```
`objectType` can be any string — common values: `"table"`, `"view"`, `"report"`,
`"dashboard"`, `"job"`, `"model"`.
`objectId` should be a stable unique identifier for the asset within the resource.
For tables, use the `fullTableId` format: `database:schema.table`.
The returned `mcon` is the stable MC identifier for this node — save it if you plan to
reference it in edges or deletions.
---
## createOrUpdateLineageEdge
Creates or updates a directed edge: source → destination (default: IS_DOWNSTREAM).
```graphql
mutation CreateOrUpdateLineageEdge(
$source: NodeInput!
$destination: NodeInput!
$expireAt: DateTime
$edgeType: EdgeType
) {
createOrUpdateLineageEdge(
source: $source
destination: $destination
expireAt: $expireAt
edgeType: $edgeType
) {
edge {
source { mcon displayName objectType }
destination { mcon displayName objectType }
isCustom
expireAt
}
}
}
```
`NodeInput` shape:
```json
{
"objectType": "table",
"objectId": "analytics:analytics.orders",
"resourceId": "<warehouse-uuid>"
}
```
**Full example — dbt model → warehouse table:**
```json
{
"source": {
"objectType": "model",
"objectId": "dbt://my_project/models/staging/stg_orders",
"resourceName": "dbt-production"
},
"destination": {
"objectType": "table",
"objectId": "analytics:analytics.orders",
"resourceId": "<snowflake-warehouse-uuid>"
},
"expireAt": "9999-12-31",
"edgeType": "IS_DOWNSTREAM"
}
```
---
## deleteLineageNode
Deletes a node and **all its edges and objects**. This is irreversible.
```graphql
mutation DeleteLineageNode($mcon: String!) {
deleteLineageNode(mcon: $mcon) {
objectsDeleted
nodesDeleted
edgesDeleted
}
}
```
Get the MCON from `createOrUpdateLineageNode`'s response, or from:
```graphql
query {
getTable(fullTableId: "analytics:analytics.orders", dwId: "<warehouse-uuid>") {
mcon
}
}
```
---
## Python helper for all three mutations
```python
import requests
GRAPHQL_URL = "https://api.getmontecarlo.com/graphql"
HEADERS = {
"x-mcd-id": "<graphql-api-key-id>",
"x-mcd-token": "<graphql-api-key-secret>",
"Content-Type": "application/json",
}
def run_mutation(query: str, variables: dict) -> dict:
resp = requests.post(GRAPHQL_URL, json={"query": query, "variables": variables}, headers=HEADERS)
resp.raise_for_status()
data = resp.json()
if "errors" in data:
raise RuntimeError(data["errors"])
return data["data"]
# Example: create a permanent node
result = run_mutation(
"""mutation($objectType: String!, $objectId: String!, $resourceId: UUID, $expireAt: DateTime) {
createOrUpdateLineageNode(objectType: $objectType, objectId: $objectId,
resourceId: $resourceId, expireAt: $expireAt) {
node { mcon displayName }
}
}""",
{
"objectType": "table",
"objectId": "analytics:analytics.orders",
"resourceId": "<warehouse-uuid>",
"expireAt": "9999-12-31",
}
)
print("MCON:", result["createOrUpdateLineageNode"]["node"]["mcon"])
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |