cianfhoghlaim/cianchosaint / .agents/skills/change-detection
change-detection
VerifiedChange Detection
When to use this skill
Use when you need to:
- "Detect when an upstream NCCA / SEC / DES page changes"
- "Wire a DLT source to re-run only when content changes"
- "Add a Dagster sensor that polls a sitemap"
- "Configure ChangeDetection.io on
arm1-oci" - "Detect when motherduck / dlthub / lancedb / cocoindex publish a new blog post"
- "Avoid the anti-pattern of polling every source every minute"
The 4-layer pattern
The Cianfhoghlaim platform detects upstream changes in 4 complementary layers. All four are running simultaneously; each one is appropriate for a different class of source.
┌─────────────────────────────────────────────────────────────┐
│ Layer 1: DLT `incremental` cursor (per-source, in-pipeline) │
│ → for paginated APIs with a monotonic id / updated_at │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ Layer 2: Dagster sitemap-hash sensor (in-Dagster) │
│ → for sources that publish a sitemap.xml │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ Layer 3: ChangeDetection.io (on arm1-oci, self-hosted) │
│ → for HTML pages without a sitemap; visual diff │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ Layer 4: Firecrawl monitor (NEW 2026-06) │
│ → for blog / changelog / docs surfaces without sitemaps, │
│ using the Firecrawl LLM-judge to filter meaningful change │
│ → powers dlt/domains/cross/ │
│ upstream/blog_post.py + the 3 v1 CocoIndex Apps │
│ (upstream_blog_monitor, upstream_api_surface, │
│ cocoindex_v1_conformance). See │
│ openspec/changes/upstream-package-monitoring. │
└─────────────────────────────────────────────────────────────┘
Layer 4 — Firecrawl monitor (NEW 2026-06)
Added by openspec/changes/upstream-package-monitoring for blog /
changelog / docs surfaces that don't publish a sitemap.xml and
where ChangeDetection.io's HTML diff is too noisy (lots of marketing
copy edits that don't matter). The Firecrawl monitor uses an LLM
judge (--goal) to filter meaningful change from noise.
Canonical recipe
# Apply a monitor (one-time):
firecrawl monitor apply \
--page https://motherduck.com/blog/ \
--goal "Alert on DuckLake / BYOB / Cortex Code changes; ignore marketing noise" \
--schedule "every 30 minutes" \
--webhook-url "https://n8n.cianfhoghlaim.ie/webhook/upstream-blog?package=motherduck" \
--retention-days 90
The 4 active monitors (canonical YAML in
infrastructure/firecrawl/monitors/upstream_packages/):
| File | Package | Goal focus |
|---|---|---|
motherduck_blog.yml | motherduck | DuckLake releases, BYOB / BYOC hosting, Cortex Code updates |
dlthub_blog.yml | dlthub | Source-context additions, ADE-Bench, Cortex Code integration |
lancedb_blog.yml | lancedb | Lance Format v2.x, Lance Blob V2, multimodal, Namespace |
cocoindex_docs.yml | cocoindex | API-surface changes (coco.App, @coco.fn, FalkorDB connector) |
Each monitor's webhook → n8n workflow
bonneagar/stacks/n8n/workflows/upstream-blog-monitor.json
(which validates the payload + writes to
s3://cianfhoghlaim-upstream-webhooks/<package>/<YYYY-MM-DD>/...jsonl)
→ DLT incremental source cianfhoghlaim.dlt.domains.cross.upstream.blog_post
→ CocoIndex v1 App upstream_blog_monitor (BAML ExtractBlogPostMetadata +
FalkorDB BlogPostNode + PackageNode + PUBLISHED_BY edges) →
Dagster asset upstream_blog_monitor_ingest.
For cocoindex docs (not a blog — a docs surface), the dedicated
upstream_api_surface CocoIndex v1 App watches 5 URLs + llms-full.txt,
BAML-extracts ApiChange records via ExtractCocoIndexApiChange, and
writes ApiChangeNode + AFFECTS_APP edges to the
upstream_packages_graph FalkorDB graph. The
upstream_breaking_change_sensor (5-min poll) fires Slack alerts to
#upstream-breaking-changes when a change_severity=high +
is_breaking=true change goes unacknowledged.
Pair this skill with: the firecrawl-cli skill (for the firecrawl monitor CLI recipe) and the cianfhoghlaim-cocoindex-v1 skill (for the
3 v1 Apps that consume the payloads).
Layer 1 — DLT incremental cursor
Every paginated API source uses dlt.sources.incremental so
the source only loads rows newer than the last successful run.
The cursor state is stored in dlt's state file
(~/.dlt/pipelines/<pipeline_name>/state.json).
import dlt
from dlt.sources.rest_api import rest_api_source
@dlt.source(name="ireland_curriculum")
def ireland_curriculum_source(
updated_at=dlt.sources.incremental(
"updated_at",
initial_value="2020-01-01T00:00:00Z",
),
):
config = {
"client": {"base_url": "https://ncca.ie/api/"},
"resources": [
{
"name": "curriculum_pages",
"endpoint": {
"path": "pages",
"params": {
"updated_since": {
"type": "incremental",
"cursor_path": "updated_at",
"initial_value": "2020-01-01T00:00:00Z",
},
},
},
},
],
}
return rest_api_source(config)
Conventions:
- Cursor field is always
updated_atorlast_modified(UTC ISO 8601) initial_valueis the project'sdltminimum (e.g."2020-01-01T00:00:00Z"for Irish sources)- For sources without a timestamp, use an incrementing
idcursor instead
Layer 2 — Dagster sitemap-hash sensor
For sources that publish a sitemap.xml, the canonical pattern
is a Dagster sensor that SHA-256-hashes the sitemap and emits a
RunRequest only when the hash changes. The project ships 5 such
sensors in orchestration/defs/sensors/:
curriculum_freshness_sensor.pydomain_sensors.pyauthor_archive_sensors.pycognee_cron_sensor.pyleabharlann_sensors.py
import hashlib
import requests
from dagster import sensor, SensorEvaluationContext, RunRequest, SkipReason
CURRICULUM_SITEMAP_URL = "https://ncca.ie/sitemap.xml"
@sensor(asset_selection=[AssetKey("ireland_curriculum_pages")])
def curriculum_sitemap_sensor(context: SensorEvaluationContext):
"""Re-run the ireland_curriculum_pages asset when the NCCA sitemap changes."""
response = requests.get(CURRICULUM_SITEMAP_URL, timeout=30)
response.raise_for_status()
current_hash = hashlib.sha256(response.content).hexdigest()
last_hash = context.cursor or ""
if current_hash == last_hash:
return SkipReason(f"NCCA sitemap unchanged (hash {current_hash[:8]})")
context.update_cursor(current_hash)
return RunRequest(run_key=current_hash[:16])
Conventions:
- The sensor returns
SkipReason(notNone) when the hash matches — Dagster UI shows the skip reason so you can see WHY the sensor did not fire - The cursor is the full SHA-256 hash (64 hex chars)
- The
run_keyis the first 16 chars of the hash (for dedup) - A polling cadence of 60s is the default; reduce to 300s for low-traffic sources
Layer 3 — ChangeDetection.io (on arm1-oci)
For HTML pages without a sitemap (or when a visual diff matters),
the project runs the ChangeDetection.io
container on arm1-oci. The Compose file is at
bonneagar/stacks/changedetection/compose.yaml.
Configuration
The sources.yaml file at bonneagar/stacks/changedetection/sources.yaml
pairs each watched URL with the Dagster job that runs when the
URL changes:
- name: "NCCA — primary curriculum (mathematics)"
url: "https://ncca.ie/en/primary/curriculum-area/mathematics"
check_interval: 3600 # 1 hour
dagster_job: "ireland_curriculum_mathematics"
dagster_asset: "ireland_curriculum_pages"
Webhook
ChangeDetection.io posts a JSON payload to a webhook
(/api/change-detected) when a source changes. The Dagster
run-launcher accepts the payload, looks up the matching asset
in sources.yaml, and emits a RunRequest for the affected
partition:
from dagster import webserver, job, RunRequest
import yaml
@webserver(path="/api/change-detected")
def change_detected_webhook(context, payload: dict):
sources = yaml.safe_load(open(SOURCES_YAML))
for src in sources:
if src["url"] == payload["url"]:
return RunRequest(run_key=payload["uuid"], asset_selection=[src["dagster_asset"]])
return {"status": "no-match"}
The sources.yaml pairing rule
Each ChangeDetection.io entry MUST declare its corresponding Dagster asset. This is the only invariant:
- name: <human-readable>
url: <full URL>
check_interval: <seconds, min 300>
dagster_job: <job name>
dagster_asset: <asset key>
If the URL is shared by multiple assets, list them all and the
webhook fans out the RunRequest to each.
Anti-patterns
- Polling every DLT source every minute — wastes API credits and warms up rate-limited sources. Use Layer 1 (incremental cursor) or Layer 2 (sitemap sensor) instead.
- ChangeDetection.io as the only mechanism — fine for one-off URLs, but for 100+ sources the UI becomes unmanageable. Use Layer 2 for the bulk, Layer 3 for the outliers.
- DLT fixed-cron without an incremental cursor — the source re-loads everything every run. Costs API credits, wastes DuckLake writes, and risks duplicates.
- firecrawl
changeTrackingJSON mode as the primary mechanism — has no history, costs credits per check, and silently misses any content that isn't in the structured output schema. - Polling at sub-minute intervals — even for fast-moving sources, 60s is the floor. Below that, switch to a webhook (the upstream pushes).
Cross-references
.agents/skills/dlt/SKILL.md— the dlt skill (Layer 1 lives here).agents/skills/dagster/SKILL.md— the Dagster skill (Layer 2 sensors live here)bonneagar/stacks/changedetection/— the Layer 3 Compose stack +sources.yamlorchestration/defs/sensors/— 5 canonical sensor implementationsdlt/british_isles/ireland/— 33+ DLT sources (all with incremental cursors)
British-Isles Education pipeline — Layer 2 wiring (post-v4)
The BIEP (openspec/changes/lc6-biep/) relies on all 4
change-detection layers for freshness. The canonical Layer 2
sensors live in orchestration/defs/sensors/:
| Sensor file | Layer | Trigger | Asset re-materialised |
|---|---|---|---|
ncca_sitemap_sensor.py | 2 | SHA-256 of ncca.ie/sitemap.xml | lc6_curriculum_syllabus (6 subjects × 2 langs) |
sec_past_paper_sensor.py | 2 | SHA-256 of examinations.ie/sitemap.xml | lc6_exam_paper_layout + lc6_marking_scheme |
gov_ie_circular_sensor.py | 3 | ChangeDetection.io on gov.ie/.../circulars/... | lc6_government_circulars (the 7th subject) |
cianfhoghlaim_lc6_daily_schedule.py | 1 + 4 | cron + Dagster schedule at 02:00 UTC | All 42 lc5/lc6 assets |
bge_m3_index_rebuild_sensor.py | 4 | Firecrawl monitor on huggingface.co/BAAI/bge-m3 | The 24+1 LanceDB tables |
British-Isles Education pipeline use case:
- 6 LC subjects — Mathematics, Chemistry, Geography,
Gaeilge, English, Computer Science, each monitored by the
ncca_sitemap_sensor.pyLayer 2 sensor with SHA-256 hash cursor. gov.iecirculars — thegov_ie_circular_sensor.pyLayer 3 sensor (ChangeDetection.io onarm1-oci) polls the gov.ie circulars pages every hour and re-materialiseslc6_government_circulars(the 7th BIEP subject).- 2 languages per subject — the
languagepartition (en/ga) is part of theMultiPartitionsDefinition, so each sensor fans out to 12 re-materialisations per SHA-256 change (6 subjects × 2 langs). bge-m3model freshness — thebge_m3_index_rebuild_sensor.pyLayer 4 sensor listens to the HuggingFace blog + release notes via Firecrawl monitor and triggers a LanceDB re-index when the canonicalBAAI/bge-m3embedder bumps a minor version.- Dagster
MultiPartitionsDefinition— the canonical Cianfhoghlaim BIEP partition:(language × subject × level)for the curriculum +(year)for the exam-papers partition.
Cross-references:
.agents/skills/dlt/SKILL.md— the DLTincrementalcursors (Layer 1) for each BIEP source.agents/skills/dagster/SKILL.md— the@dlt_assetswrappers + sensor wiring.agents/skills/cocoindex/SKILL.md— the 7 v1 Apps re-materialised by the Layer 2 sensors.agents/skills/motherduck/SKILL.md— the 4 Dives.agents/skills/secrets-management/SKILL.md— theinfisical://dev-baile/cianfhoghlaim/FIRECRAWL_API_KEY+GOV_IE_SCRAPER_TOKENsecret contract
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |