Skip to content

monte-carlo-data/mc-agent-toolkit / skills/push-ingestion

push-ingestion

Verified

Expert guide for Monte Carlo's push ingestion model. Use this skill whenever a customer or engineer mentions: pushing data to Monte Carlo, the IngestionService, pycarlo push APIs, build me a collection script, push metadata/lineage/query logs, invocation_id tracing, custom lineage nodes or edges, deleting push tables, or any question about why pushed data is not showing up. Also trigger when they ask to generate code that collects metadata, table schema, row counts, freshness, lineage, or query history from any data warehouse or data source and sends it to Monte Carlo. If the user mentions any warehouse, database, or data platform alongside any Monte Carlo topic, this skill is almost certainly relevant.

FileKindSize
README.mdmarkdown2.6 KB
SKILL.mdskill md17.9 KB
references/anomaly-detection.mdmarkdown3.8 KB
references/custom-lineage.mdmarkdown5.0 KB
references/direct-http-api.mdmarkdown5.0 KB
references/prerequisites.mdmarkdown3.6 KB
references/push-lineage.mdmarkdown5.0 KB
references/push-metadata.mdmarkdown5.4 KB
references/push-query-logs.mdmarkdown6.0 KB
references/validation.mdmarkdown4.9 KB
scripts/sample_verify.pyscript13.8 KB
scripts/templates/bigquery-iceberg/collect_and_push_metadata.pyscript2.3 KB
scripts/templates/bigquery-iceberg/collect_and_push_query_logs.pyscript2.2 KB
scripts/templates/bigquery-iceberg/collect_metadata.pyscript7.9 KB
scripts/templates/bigquery-iceberg/collect_query_logs.pyscript4.6 KB
scripts/templates/bigquery-iceberg/push_metadata.pyscript5.9 KB
scripts/templates/bigquery-iceberg/push_query_logs.pyscript6.8 KB
scripts/templates/bigquery/collect_and_push_lineage.pyscript2.5 KB
scripts/templates/bigquery/collect_and_push_metadata.pyscript2.2 KB
scripts/templates/bigquery/collect_and_push_query_logs.pyscript2.5 KB
scripts/templates/bigquery/collect_lineage.pyscript7.1 KB
scripts/templates/bigquery/collect_metadata.pyscript4.9 KB
scripts/templates/bigquery/collect_query_logs.pyscript5.5 KB
scripts/templates/bigquery/push_lineage.pyscript6.5 KB
scripts/templates/bigquery/push_metadata.pyscript6.2 KB
scripts/templates/bigquery/push_query_logs.pyscript7.2 KB
scripts/templates/databricks/collect_and_push_lineage.pyscript3.2 KB
scripts/templates/databricks/collect_and_push_metadata.pyscript3.0 KB
scripts/templates/databricks/collect_and_push_query_logs.pyscript3.4 KB
scripts/templates/databricks/collect_lineage.pyscript8.5 KB
scripts/templates/databricks/collect_metadata.pyscript7.5 KB
scripts/templates/databricks/collect_query_logs.pyscript6.9 KB
scripts/templates/databricks/push_lineage.pyscript6.5 KB
scripts/templates/databricks/push_metadata.pyscript6.0 KB
scripts/templates/databricks/push_query_logs.pyscript7.2 KB
scripts/templates/hive/collect_and_push_lineage.pyscript3.8 KB
scripts/templates/hive/collect_and_push_metadata.pyscript3.8 KB
scripts/templates/hive/collect_and_push_query_logs.pyscript3.9 KB
scripts/templates/hive/collect_lineage.pyscript9.5 KB
scripts/templates/hive/collect_metadata.pyscript10.2 KB
scripts/templates/hive/collect_query_logs.pyscript9.0 KB
scripts/templates/hive/push_lineage.pyscript10.5 KB
scripts/templates/hive/push_metadata.pyscript8.1 KB
scripts/templates/hive/push_query_logs.pyscript8.5 KB
scripts/templates/redshift/collect_and_push_lineage.pyscript3.0 KB
scripts/templates/redshift/collect_and_push_metadata.pyscript3.1 KB
scripts/templates/redshift/collect_and_push_query_logs.pyscript3.7 KB
scripts/templates/redshift/collect_lineage.pyscript7.8 KB
scripts/templates/redshift/collect_metadata.pyscript7.1 KB
scripts/templates/redshift/collect_query_logs.pyscript8.4 KB
scripts/templates/redshift/push_lineage.pyscript6.1 KB
scripts/templates/redshift/push_metadata.pyscript5.9 KB
scripts/templates/redshift/push_query_logs.pyscript7.0 KB
scripts/templates/snowflake/collect_and_push_lineage.pyscript4.8 KB
scripts/templates/snowflake/collect_and_push_metadata.pyscript4.3 KB
scripts/templates/snowflake/collect_and_push_query_logs.pyscript4.3 KB
scripts/templates/snowflake/collect_lineage.pyscript11.6 KB
scripts/templates/snowflake/collect_metadata.pyscript10.7 KB
scripts/templates/snowflake/collect_query_logs.pyscript8.1 KB
scripts/templates/snowflake/push_lineage.pyscript10.1 KB
scripts/templates/snowflake/push_metadata.pyscript7.1 KB
scripts/templates/snowflake/push_query_logs.pyscript8.3 KB
scripts/test_template_sdk_usage.pyscript9.1 KB

scripts/templates/bigquery/collect_metadata.py

View on GitHub
"""
BigQuery — Metadata Collection (collect only)
==============================================
Collects table schemas, row counts, byte sizes, and descriptions from all
datasets in a BigQuery project and writes them to a JSON manifest file.

Can be run standalone via CLI or imported (use the ``collect()`` function).

Substitution points (search for "← SUBSTITUTE"):
  - BIGQUERY_PROJECT_ID   : GCP project ID to collect from
  - GOOGLE_APPLICATION_CREDENTIALS : path to service-account JSON key file
  - DATASET_EXCLUSIONS    : datasets to skip (informational / system datasets)

Prerequisites:
  pip install google-cloud-bigquery
"""

from __future__ import annotations

import argparse
import json
import logging
import os
from datetime import datetime, timezone

from google.cloud import bigquery

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)

RESOURCE_TYPE = "bigquery"

# Datasets to skip — add any internal / system datasets here
DATASET_EXCLUSIONS = {  # ← SUBSTITUTE: add datasets to exclude
    "_bqc_",
    "INFORMATION_SCHEMA",
}

# BigQuery type → Monte Carlo canonical type
BQ_TYPE_MAP: dict[str, str] = {
    "INT64": "INTEGER",
    "INTEGER": "INTEGER",
    "FLOAT64": "FLOAT",
    "FLOAT": "FLOAT",
    "BOOL": "BOOLEAN",
    "BOOLEAN": "BOOLEAN",
    "STRING": "VARCHAR",
    "BYTES": "BINARY",
    "DATE": "DATE",
    "DATETIME": "DATETIME",
    "TIMESTAMP": "TIMESTAMP",
    "TIME": "TIME",
    "NUMERIC": "DECIMAL",
    "BIGNUMERIC": "DECIMAL",
    "RECORD": "STRUCT",
    "STRUCT": "STRUCT",
    "REPEATED": "ARRAY",
    "JSON": "JSON",
    "GEOGRAPHY": "GEOGRAPHY",
}


def map_bq_type(bq_type: str) -> str:
    return BQ_TYPE_MAP.get(bq_type.upper(), bq_type.upper())


def _collect_assets(bq_client: bigquery.Client, project_id: str) -> list[dict]:
    """Collect table metadata from BigQuery and return as a list of dicts."""
    assets: list[dict] = []

    for dataset_item in bq_client.list_datasets():
        dataset_id = dataset_item.dataset_id

        if any(exc in dataset_id for exc in DATASET_EXCLUSIONS):
            log.info("Skipping dataset %s", dataset_id)
            continue

        dataset_ref = bq_client.dataset(dataset_id)

        for table_item in bq_client.list_tables(dataset_ref):
            table_ref = dataset_ref.table(table_item.table_id)
            table = bq_client.get_table(table_ref)

            fields = [
                {
                    "name": field.name,
                    "type": map_bq_type(field.field_type),
                    "description": field.description or None,
                }
                for field in table.schema
            ]

            asset = {
                "name": table.table_id,
                "database": project_id,  # ← SUBSTITUTE: use project or dataset as database
                "schema": dataset_id,
                "type": "VIEW" if table.table_type == "VIEW" else "TABLE",
                "description": table.description or None,
                "fields": fields,
                "volume": {
                    "row_count": table.num_rows,
                    "byte_count": table.num_bytes,
                },
                "freshness": {
                    "last_updated_time": table.modified.isoformat() if table.modified else None,
                },
            }
            assets.append(asset)
            log.info("Queued %s.%s.%s", project_id, dataset_id, table.table_id)

    return assets


def collect(
    project_id: str,
    output_file: str = "metadata_output.json",
) -> dict:
    """
    Connect to BigQuery, collect table metadata, and write a JSON manifest.

    Returns the manifest dict.
    """
    bq_client = bigquery.Client(project=project_id)  # ← SUBSTITUTE: adjust auth if needed

    log.info("Collecting metadata from project %s ...", project_id)
    assets = _collect_assets(bq_client, project_id)
    log.info("Collected %d asset(s).", len(assets))

    manifest = {
        "resource_type": RESOURCE_TYPE,
        "collected_at": datetime.now(timezone.utc).isoformat(),
        "assets": assets,
    }
    with open(output_file, "w") as fh:
        json.dump(manifest, fh, indent=2)
    log.info("Asset manifest written to %s", output_file)

    return manifest


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Collect BigQuery metadata and write to a manifest file",
    )
    parser.add_argument("--project-id", default=os.getenv("BIGQUERY_PROJECT_ID"))  # ← SUBSTITUTE
    parser.add_argument("--output-file", default="metadata_output.json")
    args = parser.parse_args()

    missing = [k for k, v in vars(args).items() if v is None and k != "output_file"]
    if missing:
        parser.error(f"Missing required arguments/env vars: {missing}")

    collect(
        project_id=args.project_id,
        output_file=args.output_file,
    )


if __name__ == "__main__":
    main()
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall