Skip to content

monte-carlo-data/mc-agent-toolkit / skills/push-ingestion

push-ingestion

Verified

Expert guide for Monte Carlo's push ingestion model. Use this skill whenever a customer or engineer mentions: pushing data to Monte Carlo, the IngestionService, pycarlo push APIs, build me a collection script, push metadata/lineage/query logs, invocation_id tracing, custom lineage nodes or edges, deleting push tables, or any question about why pushed data is not showing up. Also trigger when they ask to generate code that collects metadata, table schema, row counts, freshness, lineage, or query history from any data warehouse or data source and sends it to Monte Carlo. If the user mentions any warehouse, database, or data platform alongside any Monte Carlo topic, this skill is almost certainly relevant.

FileKindSize
README.mdmarkdown2.6 KB
SKILL.mdskill md17.9 KB
references/anomaly-detection.mdmarkdown3.8 KB
references/custom-lineage.mdmarkdown5.0 KB
references/direct-http-api.mdmarkdown5.0 KB
references/prerequisites.mdmarkdown3.6 KB
references/push-lineage.mdmarkdown5.0 KB
references/push-metadata.mdmarkdown5.4 KB
references/push-query-logs.mdmarkdown6.0 KB
references/validation.mdmarkdown4.9 KB
scripts/sample_verify.pyscript13.8 KB
scripts/templates/bigquery-iceberg/collect_and_push_metadata.pyscript2.3 KB
scripts/templates/bigquery-iceberg/collect_and_push_query_logs.pyscript2.2 KB
scripts/templates/bigquery-iceberg/collect_metadata.pyscript7.9 KB
scripts/templates/bigquery-iceberg/collect_query_logs.pyscript4.6 KB
scripts/templates/bigquery-iceberg/push_metadata.pyscript5.9 KB
scripts/templates/bigquery-iceberg/push_query_logs.pyscript6.8 KB
scripts/templates/bigquery/collect_and_push_lineage.pyscript2.5 KB
scripts/templates/bigquery/collect_and_push_metadata.pyscript2.2 KB
scripts/templates/bigquery/collect_and_push_query_logs.pyscript2.5 KB
scripts/templates/bigquery/collect_lineage.pyscript7.1 KB
scripts/templates/bigquery/collect_metadata.pyscript4.9 KB
scripts/templates/bigquery/collect_query_logs.pyscript5.5 KB
scripts/templates/bigquery/push_lineage.pyscript6.5 KB
scripts/templates/bigquery/push_metadata.pyscript6.2 KB
scripts/templates/bigquery/push_query_logs.pyscript7.2 KB
scripts/templates/databricks/collect_and_push_lineage.pyscript3.2 KB
scripts/templates/databricks/collect_and_push_metadata.pyscript3.0 KB
scripts/templates/databricks/collect_and_push_query_logs.pyscript3.4 KB
scripts/templates/databricks/collect_lineage.pyscript8.5 KB
scripts/templates/databricks/collect_metadata.pyscript7.5 KB
scripts/templates/databricks/collect_query_logs.pyscript6.9 KB
scripts/templates/databricks/push_lineage.pyscript6.5 KB
scripts/templates/databricks/push_metadata.pyscript6.0 KB
scripts/templates/databricks/push_query_logs.pyscript7.2 KB
scripts/templates/hive/collect_and_push_lineage.pyscript3.8 KB
scripts/templates/hive/collect_and_push_metadata.pyscript3.8 KB
scripts/templates/hive/collect_and_push_query_logs.pyscript3.9 KB
scripts/templates/hive/collect_lineage.pyscript9.5 KB
scripts/templates/hive/collect_metadata.pyscript10.2 KB
scripts/templates/hive/collect_query_logs.pyscript9.0 KB
scripts/templates/hive/push_lineage.pyscript10.5 KB
scripts/templates/hive/push_metadata.pyscript8.1 KB
scripts/templates/hive/push_query_logs.pyscript8.5 KB
scripts/templates/redshift/collect_and_push_lineage.pyscript3.0 KB
scripts/templates/redshift/collect_and_push_metadata.pyscript3.1 KB
scripts/templates/redshift/collect_and_push_query_logs.pyscript3.7 KB
scripts/templates/redshift/collect_lineage.pyscript7.8 KB
scripts/templates/redshift/collect_metadata.pyscript7.1 KB
scripts/templates/redshift/collect_query_logs.pyscript8.4 KB
scripts/templates/redshift/push_lineage.pyscript6.1 KB
scripts/templates/redshift/push_metadata.pyscript5.9 KB
scripts/templates/redshift/push_query_logs.pyscript7.0 KB
scripts/templates/snowflake/collect_and_push_lineage.pyscript4.8 KB
scripts/templates/snowflake/collect_and_push_metadata.pyscript4.3 KB
scripts/templates/snowflake/collect_and_push_query_logs.pyscript4.3 KB
scripts/templates/snowflake/collect_lineage.pyscript11.6 KB
scripts/templates/snowflake/collect_metadata.pyscript10.7 KB
scripts/templates/snowflake/collect_query_logs.pyscript8.1 KB
scripts/templates/snowflake/push_lineage.pyscript10.1 KB
scripts/templates/snowflake/push_metadata.pyscript7.1 KB
scripts/templates/snowflake/push_query_logs.pyscript8.3 KB
scripts/test_template_sdk_usage.pyscript9.1 KB

references/direct-http-api.md

View on GitHub
# Direct HTTP API (without pycarlo)

The `pycarlo` SDK is optional. You can call the push APIs directly over HTTPS from any
language or tool (curl, Postman, etc.) as long as you:
- authenticate with an integration key whose scope is `Ingestion`
- send a JSON body that matches the ingest schema
- send to the correct integration gateway endpoint

## Endpoint

The host is environment-specific:
- **Production**: `https://integrations.getmontecarlo.com`

## Authentication headers

All requests use the same headers:
```
x-mcd-id:       <integration-key-id>
x-mcd-token:    <integration-key-secret>
Content-Type:   application/json
```

## Response

On success, all endpoints return:
```json
{"invocation_id": "<uuid>"}
```

Save the `invocation_id` — it is the primary trace ID for debugging across downstream systems.

---

## Metadata example

`POST /ingest/v1/metadata`

```bash
curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/metadata" \
  -H "Content-Type: application/json" \
  -H "x-mcd-id: <integration-key-id>" \
  -H "x-mcd-token: <integration-key-secret>" \
  -d '{
    "event_type": "RELATIONAL_ASSET",
    "resource": {
      "uuid": "<warehouse-uuid>",
      "resource_type": "snowflake"
    },
    "events": [
      {
        "type": "TABLE",
        "metadata": {
          "name": "orders",
          "database": "analytics",
          "schema": "public",
          "description": "Orders table"
        },
        "fields": [
          {"name": "id", "type": "INTEGER"},
          {"name": "amount", "type": "DECIMAL(10,2)"}
        ],
        "volume": {
          "row_count": 1000000,
          "byte_count": 111111111
        },
        "freshness": {
          "last_update_time": "2026-03-12T14:30:00Z"
        }
      }
    ]
  }'
```

`volume` and `freshness` are optional — you can push schema-only metadata.

---

## Table lineage example

`POST /ingest/v1/lineage` with `event_type: "LINEAGE"`

```bash
curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/lineage" \
  -H "Content-Type: application/json" \
  -H "x-mcd-id: <integration-key-id>" \
  -H "x-mcd-token: <integration-key-secret>" \
  -d '{
    "event_type": "LINEAGE",
    "resource": {
      "uuid": "<warehouse-uuid>",
      "resource_type": "snowflake"
    },
    "events": [
      {
        "source": {
          "name": "orders_raw",
          "database": "analytics",
          "schema": "public"
        },
        "destination": {
          "name": "orders_curated",
          "database": "analytics",
          "schema": "public"
        }
      }
    ]
  }'
```

---

## Column lineage example

`POST /ingest/v1/lineage` with `event_type: "COLUMN_LINEAGE"`

Same endpoint as table lineage. Column lineage automatically creates the parent table-level
edge too.

```bash
curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/lineage" \
  -H "Content-Type: application/json" \
  -H "x-mcd-id: <integration-key-id>" \
  -H "x-mcd-token: <integration-key-secret>" \
  -d '{
    "event_type": "COLUMN_LINEAGE",
    "resource": {
      "uuid": "<warehouse-uuid>",
      "resource_type": "snowflake"
    },
    "events": [
      {
        "source": {
          "name": "customers",
          "database": "analytics",
          "schema": "public"
        },
        "destination": {
          "name": "customer_orders",
          "database": "analytics",
          "schema": "public"
        },
        "col_mappings": [
          {
            "destination_col": "customer_id",
            "source_cols": ["customer_id"]
          },
          {
            "destination_col": "full_name",
            "source_cols": ["first_name", "last_name"]
          }
        ]
      }
    ]
  }'
```

---

## Query log example

`POST /ingest/v1/querylogs`

**Important**: this endpoint uses `log_type` instead of `resource_type` in the resource object.
This is the only endpoint where the field name differs.

```bash
curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/querylogs" \
  -H "Content-Type: application/json" \
  -H "x-mcd-id: <integration-key-id>" \
  -H "x-mcd-token: <integration-key-secret>" \
  -d '{
    "event_type": "QUERY_LOG",
    "resource": {
      "uuid": "<warehouse-uuid>",
      "log_type": "snowflake"
    },
    "events": [
      {
        "start_time": "2026-03-02T12:00:00Z",
        "end_time": "2026-03-02T12:00:05Z",
        "query_text": "SELECT * FROM analytics.public.orders",
        "query_id": "query-123",
        "user": "analyst@company.com",
        "returned_rows": 10
      }
    ]
  }'
```

Supported `log_type` values: `snowflake`, `bigquery`, `databricks`, `redshift`, `hive-s3`,
`athena`, `teradata`, `clickhouse`, `databricks-metastore-sql-warehouse`, `s3`, `presto-s3`.

---

## Batching

The compressed request body must not exceed **1MB** (Kinesis limit). For large payloads, split
events into multiple requests. Each request returns its own `invocation_id`.

## Expiration summary

| Flow | Expiration |
|---|---|
| Table metadata | Never expires |
| Table lineage | Never expires |
| Column lineage | Expires after 10 days |
| Query logs | Same as pulled query logs |
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall