Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/change-detection

change-detection

Verified

FileKindSize
SKILL.mdskill md14.7 KB
---
name: change-detection
description: How the Cianfhoghlaim platform detects upstream changes for `dlt/sources.yaml`. Use when writing a DLT source that needs to re-run on upstream change, wiring a Dagster sensor, or configuring ChangeDetection.io on `arm1-oci`. Covers the 4-layer pattern: DLT incremental cursor + Dagster sitemap-hash sensor + ChangeDetection.io + Firecrawl monitor (the 4th layer, added 2026-06 for blog/changelog-without-sitemap surfaces). Powers BIEP freshness for the 6 LC subjects (Mathematics, Chemistry, Geography, Gaeilge, English, Computer Science) + `gov.ie` education circulars.
---

# Change Detection

## When to use this skill

Use when you need to:

- "Detect when an upstream NCCA / SEC / DES page changes"
- "Wire a DLT source to re-run only when content changes"
- "Add a Dagster sensor that polls a sitemap"
- "Configure ChangeDetection.io on `arm1-oci`"
- "Detect when motherduck / dlthub / lancedb / cocoindex publish a new blog post"
- "Avoid the anti-pattern of polling every source every minute"

## The 4-layer pattern

The Cianfhoghlaim platform detects upstream changes in 4
complementary layers. **All four are running simultaneously**;
each one is appropriate for a different class of source.

```
┌─────────────────────────────────────────────────────────────┐
│  Layer 1: DLT `incremental` cursor (per-source, in-pipeline) │
│  → for paginated APIs with a monotonic id / updated_at       │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│  Layer 2: Dagster sitemap-hash sensor (in-Dagster)            │
│  → for sources that publish a sitemap.xml                     │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│  Layer 3: ChangeDetection.io (on arm1-oci, self-hosted)      │
│  → for HTML pages without a sitemap; visual diff             │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│  Layer 4: Firecrawl monitor (NEW 2026-06)                     │
│  → for blog / changelog / docs surfaces without sitemaps,     │
│    using the Firecrawl LLM-judge to filter meaningful change  │
│    → powers dlt/domains/cross/                  │
│      upstream/blog_post.py + the 3 v1 CocoIndex Apps          │
│      (upstream_blog_monitor, upstream_api_surface,            │
│       cocoindex_v1_conformance). See                          │
│      openspec/changes/upstream-package-monitoring.             │
└─────────────────────────────────────────────────────────────┘
```

## Layer 4 — Firecrawl monitor (NEW 2026-06)

Added by `openspec/changes/upstream-package-monitoring` for blog /
changelog / docs surfaces that don't publish a `sitemap.xml` and
where ChangeDetection.io's HTML diff is too noisy (lots of marketing
copy edits that don't matter). The Firecrawl monitor uses an LLM
judge (`--goal`) to filter meaningful change from noise.

### Canonical recipe

```bash
# Apply a monitor (one-time):
firecrawl monitor apply \
  --page https://motherduck.com/blog/ \
  --goal "Alert on DuckLake / BYOB / Cortex Code changes; ignore marketing noise" \
  --schedule "every 30 minutes" \
  --webhook-url "https://n8n.cianfhoghlaim.ie/webhook/upstream-blog?package=motherduck" \
  --retention-days 90
```

The 4 active monitors (canonical YAML in
`infrastructure/firecrawl/monitors/upstream_packages/`):

| File | Package | Goal focus |
|:--|:--|:--|
| `motherduck_blog.yml` | motherduck | DuckLake releases, BYOB / BYOC hosting, Cortex Code updates |
| `dlthub_blog.yml` | dlthub | Source-context additions, ADE-Bench, Cortex Code integration |
| `lancedb_blog.yml` | lancedb | Lance Format v2.x, Lance Blob V2, multimodal, Namespace |
| `cocoindex_docs.yml` | cocoindex | API-surface changes (coco.App, @coco.fn, FalkorDB connector) |

Each monitor's webhook → n8n workflow
`bonneagar/stacks/n8n/workflows/upstream-blog-monitor.json`
(which validates the payload + writes to
`s3://cianfhoghlaim-upstream-webhooks/<package>/<YYYY-MM-DD>/...jsonl`)
→ DLT incremental source `cianfhoghlaim.dlt.domains.cross.upstream.blog_post`
→ CocoIndex v1 App `upstream_blog_monitor` (BAML `ExtractBlogPostMetadata` +
FalkorDB `BlogPostNode` + `PackageNode` + `PUBLISHED_BY` edges) →
Dagster asset `upstream_blog_monitor_ingest`.

For cocoindex docs (not a blog — a docs surface), the dedicated
`upstream_api_surface` CocoIndex v1 App watches 5 URLs + `llms-full.txt`,
BAML-extracts `ApiChange` records via `ExtractCocoIndexApiChange`, and
writes `ApiChangeNode` + `AFFECTS_APP` edges to the
`upstream_packages_graph` FalkorDB graph. The
`upstream_breaking_change_sensor` (5-min poll) fires Slack alerts to
`#upstream-breaking-changes` when a `change_severity=high` +
`is_breaking=true` change goes unacknowledged.

**Pair this skill with**: the `firecrawl-cli` skill (for the `firecrawl
monitor` CLI recipe) and the `cianfhoghlaim-cocoindex-v1` skill (for the
3 v1 Apps that consume the payloads).

## Layer 1 — DLT `incremental` cursor

Every paginated API source uses `dlt.sources.incremental` so
the source only loads rows newer than the last successful run.
The cursor state is stored in dlt's state file
(`~/.dlt/pipelines/<pipeline_name>/state.json`).

```python
import dlt
from dlt.sources.rest_api import rest_api_source

@dlt.source(name="ireland_curriculum")
def ireland_curriculum_source(
    updated_at=dlt.sources.incremental(
        "updated_at",
        initial_value="2020-01-01T00:00:00Z",
    ),
):
    config = {
        "client": {"base_url": "https://ncca.ie/api/"},
        "resources": [
            {
                "name": "curriculum_pages",
                "endpoint": {
                    "path": "pages",
                    "params": {
                        "updated_since": {
                            "type": "incremental",
                            "cursor_path": "updated_at",
                            "initial_value": "2020-01-01T00:00:00Z",
                        },
                    },
                },
            },
        ],
    }
    return rest_api_source(config)
```

**Conventions:**

- Cursor field is **always** `updated_at` or `last_modified` (UTC
  ISO 8601)
- `initial_value` is the project's `dlt` minimum (e.g.
  `"2020-01-01T00:00:00Z"` for Irish sources)
- For sources without a timestamp, use an incrementing `id`
  cursor instead

## Layer 2 — Dagster sitemap-hash sensor

For sources that publish a `sitemap.xml`, the canonical pattern
is a Dagster sensor that SHA-256-hashes the sitemap and emits a
`RunRequest` only when the hash changes. The project ships 5 such
sensors in `orchestration/defs/sensors/`:

- `curriculum_freshness_sensor.py`
- `domain_sensors.py`
- `author_archive_sensors.py`
- `cognee_cron_sensor.py`
- `leabharlann_sensors.py`

```python
import hashlib
import requests
from dagster import sensor, SensorEvaluationContext, RunRequest, SkipReason

CURRICULUM_SITEMAP_URL = "https://ncca.ie/sitemap.xml"

@sensor(asset_selection=[AssetKey("ireland_curriculum_pages")])
def curriculum_sitemap_sensor(context: SensorEvaluationContext):
    """Re-run the ireland_curriculum_pages asset when the NCCA sitemap changes."""
    response = requests.get(CURRICULUM_SITEMAP_URL, timeout=30)
    response.raise_for_status()
    current_hash = hashlib.sha256(response.content).hexdigest()

    last_hash = context.cursor or ""
    if current_hash == last_hash:
        return SkipReason(f"NCCA sitemap unchanged (hash {current_hash[:8]})")

    context.update_cursor(current_hash)
    return RunRequest(run_key=current_hash[:16])
```

**Conventions:**

- The sensor returns `SkipReason` (not `None`) when the hash
  matches — Dagster UI shows the skip reason so you can see WHY
  the sensor did not fire
- The cursor is the full SHA-256 hash (64 hex chars)
- The `run_key` is the first 16 chars of the hash (for dedup)
- A polling cadence of 60s is the default; reduce to 300s for
  low-traffic sources

## Layer 3 — ChangeDetection.io (on `arm1-oci`)

For HTML pages without a sitemap (or when a visual diff matters),
the project runs the [ChangeDetection.io](https://changedetection.io)
container on `arm1-oci`. The Compose file is at
`bonneagar/stacks/changedetection/compose.yaml`.

### Configuration

The `sources.yaml` file at `bonneagar/stacks/changedetection/sources.yaml`
pairs each watched URL with the Dagster job that runs when the
URL changes:

```yaml
- name: "NCCA — primary curriculum (mathematics)"
  url: "https://ncca.ie/en/primary/curriculum-area/mathematics"
  check_interval: 3600  # 1 hour
  dagster_job: "ireland_curriculum_mathematics"
  dagster_asset: "ireland_curriculum_pages"
```

### Webhook

ChangeDetection.io posts a JSON payload to a webhook
(`/api/change-detected`) when a source changes. The Dagster
run-launcher accepts the payload, looks up the matching asset
in `sources.yaml`, and emits a `RunRequest` for the affected
partition:

```python
from dagster import webserver, job, RunRequest
import yaml

@webserver(path="/api/change-detected")
def change_detected_webhook(context, payload: dict):
    sources = yaml.safe_load(open(SOURCES_YAML))
    for src in sources:
        if src["url"] == payload["url"]:
            return RunRequest(run_key=payload["uuid"], asset_selection=[src["dagster_asset"]])
    return {"status": "no-match"}
```

## The `sources.yaml` pairing rule

Each ChangeDetection.io entry MUST declare its corresponding
Dagster asset. This is the only invariant:

```yaml
- name: <human-readable>
  url: <full URL>
  check_interval: <seconds, min 300>
  dagster_job: <job name>
  dagster_asset: <asset key>
```

If the URL is shared by multiple assets, list them all and the
webhook fans out the `RunRequest` to each.

## Anti-patterns

- **Polling every DLT source every minute** — wastes API credits
  and warms up rate-limited sources. Use Layer 1 (incremental
  cursor) or Layer 2 (sitemap sensor) instead.
- **ChangeDetection.io as the only mechanism** — fine for
  one-off URLs, but for 100+ sources the UI becomes
  unmanageable. Use Layer 2 for the bulk, Layer 3 for the
  outliers.
- **DLT fixed-cron without an incremental cursor** — the source
  re-loads everything every run. Costs API credits, wastes
  DuckLake writes, and risks duplicates.
- **firecrawl `changeTracking` JSON mode as the primary
  mechanism** — has no history, costs credits per check, and
  silently misses any content that isn't in the structured
  output schema.
- **Polling at sub-minute intervals** — even for fast-moving
  sources, 60s is the floor. Below that, switch to a webhook
  (the upstream pushes).

## Cross-references

- `.agents/skills/dlt/SKILL.md` — the dlt skill (Layer 1 lives
  here)
- `.agents/skills/dagster/SKILL.md` — the Dagster skill
  (Layer 2 sensors live here)
- `bonneagar/stacks/changedetection/` — the Layer 3
  Compose stack + `sources.yaml`
- `orchestration/defs/sensors/` — 5 canonical sensor
  implementations
- `dlt/british_isles/ireland/` — 33+ DLT sources (all
  with incremental cursors)

## British-Isles Education pipeline — Layer 2 wiring (post-v4)

The BIEP (`openspec/changes/lc6-biep/`) relies on all 4
change-detection layers for freshness. The canonical Layer 2
sensors live in `orchestration/defs/sensors/`:

| Sensor file | Layer | Trigger | Asset re-materialised |
|:--|:--|:--|:--|
| `ncca_sitemap_sensor.py` | 2 | SHA-256 of `ncca.ie/sitemap.xml` | `lc6_curriculum_syllabus` (6 subjects × 2 langs) |
| `sec_past_paper_sensor.py` | 2 | SHA-256 of `examinations.ie/sitemap.xml` | `lc6_exam_paper_layout` + `lc6_marking_scheme` |
| `gov_ie_circular_sensor.py` | 3 | ChangeDetection.io on `gov.ie/.../circulars/...` | `lc6_government_circulars` (the 7th subject) |
| `cianfhoghlaim_lc6_daily_schedule.py` | 1 + 4 | cron + Dagster schedule at 02:00 UTC | All 42 lc5/lc6 assets |
| `bge_m3_index_rebuild_sensor.py` | 4 | Firecrawl monitor on `huggingface.co/BAAI/bge-m3` | The 24+1 LanceDB tables |

**British-Isles Education pipeline use case:**

- **6 LC subjects** — Mathematics, Chemistry, Geography,
  Gaeilge, English, Computer Science, each monitored by the
  `ncca_sitemap_sensor.py` Layer 2 sensor with SHA-256 hash
  cursor.
- **`gov.ie` circulars** — the `gov_ie_circular_sensor.py`
  Layer 3 sensor (ChangeDetection.io on `arm1-oci`) polls the
  gov.ie circulars pages every hour and re-materialises
  `lc6_government_circulars` (the 7th BIEP subject).
- **2 languages per subject** — the `language` partition
  (`en` / `ga`) is part of the `MultiPartitionsDefinition`, so
  each sensor fans out to 12 re-materialisations per SHA-256
  change (6 subjects × 2 langs).
- **`bge-m3` model freshness** — the
  `bge_m3_index_rebuild_sensor.py` Layer 4 sensor listens to
  the HuggingFace blog + release notes via Firecrawl monitor
  and triggers a LanceDB re-index when the canonical
  `BAAI/bge-m3` embedder bumps a minor version.
- **Dagster `MultiPartitionsDefinition`** — the canonical
  Cianfhoghlaim BIEP partition: `(language × subject × level)` for the
  curriculum + `(year)` for the exam-papers partition.

Cross-references:
- [`.agents/skills/dlt/SKILL.md`](../dlt/SKILL.md) — the DLT
  `incremental` cursors (Layer 1) for each BIEP source
- [`.agents/skills/dagster/SKILL.md`](../dagster/SKILL.md) —
  the `@dlt_assets` wrappers + sensor wiring
- [`.agents/skills/cocoindex/SKILL.md`](../cocoindex/SKILL.md) —
  the 7 v1 Apps re-materialised by the Layer 2 sensors
- [`.agents/skills/motherduck/SKILL.md`](../motherduck/SKILL.md) —
  the 4 Dives
- [`.agents/skills/secrets-management/SKILL.md`](../secrets-management/SKILL.md) —
  the `infisical://dev-baile/cianfhoghlaim/FIRECRAWL_API_KEY` +
  `GOV_IE_SCRAPER_TOKEN` secret contract
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall