cianfhoghlaim/cianchosaint / .agents/skills/dlt
dlt
Verified| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 20.6 KB |
| references/crawl4ai-dlt-summary.md | markdown | 1.8 KB |
| references/dagster-dlt-assets.md | markdown | 3.9 KB |
| references/deploy-gcp-cloud-function-webhook.md | markdown | 2.1 KB |
| references/deploy-gcp-cloud-function.md | markdown | 2.0 KB |
| references/deploy-gcp-cloud-run.md | markdown | 2.4 KB |
| references/deploy-modal.md | markdown | 2.6 KB |
| references/destinations-cognee-memgraph.md | markdown | 3.3 KB |
| references/destinations-graphiti.md | markdown | 3.0 KB |
| references/destinations-lancedb.md | markdown | 3.4 KB |
| references/dlt-baml-orpc-mcp-typesafe-pipeline-analysis.md | markdown | 20.0 KB |
| references/dlt-transformations.md | markdown | 3.3 KB |
| references/dlthub-codebase-analysis.md | markdown | 32.4 KB |
| references/dlthub.md | markdown | 12.5 KB |
| references/openapi-generator.md | markdown | 2.3 KB |
| references/performance-optimisation.md | markdown | 3.3 KB |
| references/sqlmesh-init.md | markdown | 1.7 KB |
| references/type-safe-pipeline.md | markdown | 4.4 KB |
references/crawl4ai-dlt-summary.md
View on GitHub# crawl4ai → dlt Summary
**crawl4ai** is a sister scraper to Firecrawl with stronger
JavaScript-rendering support. It's useful for sites that don't
work well with the standard `FirecrawlSource` (e.g. SPAs that
require JS execution to render content).
## Pattern
```python
import dlt
import crawl4ai
@dlt.resource(name="scraped_pages")
def scraped_pages(urls: list[str]):
for url in urls:
# crawl4ai handles JS rendering, proxy rotation, etc.
result = crawl4ai.run(url)
yield {
"url": url,
"title": result.metadata.get("title"),
"content": result.markdown,
"scraped_at": result.scraped_at,
}
pipeline = dlt.pipeline(destination="duckdb", dataset_name="scraped")
load_info = pipeline.run(scraped_pages(["https://example.com/page1", ...]))
```
## When to use crawl4ai vs Firecrawl
| | crawl4ai | Firecrawl |
|:--|:--|:--|
| **JS rendering** | Excellent (Playwright-based) | Good (headless browser) |
| **Speed** | Slower (full browser) | Faster (HTTP-first, JS on demand) |
| **Cost** | Free (self-hosted) | API credits |
| **Proxy rotation** | Built-in | Optional |
| **Best for** | SPAs, JS-heavy sites | Static + light JS sites |
| **Cianfhoghlaim usage** | Rare (Firecrawl is primary) | Primary (NCCA, SEC, DES) |
## Cianfhoghlaim usage
- The Cianfhoghlaim stack uses **Firecrawl** as the primary scraper
(`FirecrawlSource` in `dlt/`)
- `crawl4ai` is the fallback for JS-heavy sites that Firecrawl
cannot handle
- The `cianfhoghlaim-leabharlann` spec uses crawl4ai for some Google
Takeout HTML pages
## Reference
- The `crawl4ai-dlt.md` reference (80K, the full crawl4ai → dlt
integration) was in `docs/dlt/` (deleted with the
`sync-skills-from-docs` change)
- The `crawl4ai` skill for the upstream crawl4ai patterns
- The `firecrawl` skill for the Firecrawl alternative
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |