diffbot/diffbot-skills / skills/diffbot-crawl
diffbot-crawl
VerifiedCrawls a website from a seed URL and extracts every page it finds into markdown or typed JSON. Prefer when a task needs to discover a catalog of pages from an entire site, or deep search a site for structured data (e.g. all product inventory, all press releases, PDFs). Crawls run on Diffbot's servers; the CLI streams each page as it is processed and can list or delete jobs. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl, every page on, all pages of, whole site.
Diffbot Crawler
Crawl websites for structured content via the Diffbot Crawlbot API, and manage crawler jobs.
The db crawl* CLI commands
Always invoke with the fixed path ~/.diffbot/venv/bin/db.
~/.diffbot/venv/bin/db crawl <SITE> [options] # start a crawl and stream events
~/.diffbot/venv/bin/db crawl-list-jobs [JOB_NAME] # list all jobs or inspect one
~/.diffbot/venv/bin/db crawl-delete-job <JOB_NAME> # delete a job
db crawl options
| Flag | Default | Description |
|---|---|---|
--hops | 2 | Maximum link depth from seed URLs |
--job-name | auto-generated | Name for the crawler job |
--max-to-crawl | 100 | Maximum pages to crawl |
--max-to-process | 100 | Maximum pages to process |
--restrict-domain | true | Only follow links on the same domain as seeds |
--api-url | "" | Diffbot API endpoint for processing (default: auto/analyze) |
--crawl-delay | -1 | Delay between requests to same domain (seconds) |
--url-crawl-pattern | none | Only crawl URLs matching this pattern |
--url-process-pattern | none | Only process URLs matching this pattern |
--obey-robots | off | Obey robots.txt |
--use-proxies | off | Use proxies for crawling |
--custom-headers | none | Newline-separated custom HTTP headers |
-f, --format | markdown | Output: markdown or json |
-o, --output | stdout | Write output to file |
Workflow
Step 1 — check the CLI
~/.diffbot/venv/bin/db --version
The db CLI is a one-time user install at ~/.diffbot/venv/bin/db, the same way gh or
wrangler is — this skill never installs, upgrades, or uninstalls it. If --version
fails with No such file or directory, stop and show the user this line to run or approve
themselves (it is deliberately not pre-approved):
python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install 'diffbot>=3.4.0,<4'
If it prints a version below 3.4.0 (including 3.1.x and 3.2.x — upgrade them), fails with No such option '--version' (an old
diffbot-python install, which predates the flag), or warns about the diffbot-python
rename, point the user at Upgrading the db CLI in the plugin README and stop. Never run an install,
upgrade, or uninstall line yourself. 3.1.0 added --version and the rename; 3.2.0 switched DQL to POST; 3.4.0 is the release these skills require.
Save output files under .diffbot/tmp/ in the working directory. db creates that folder itself, so don't mkdir
it first. db, jq, and cat are not pre-approved — Claude will ask before running them. Always tell the user where an export was saved.
The token is read from DIFFBOT_API_TOKEN env var or ~/.diffbot/credentials. If missing:
echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials
Step 2 — crawl a site
Basic crawl with defaults (100 pages, 2 hops deep):
~/.diffbot/venv/bin/db crawl "https://example.com"
Deep crawl of a documentation site, saving to JSON:
~/.diffbot/venv/bin/db crawl "https://docs.example.com" \
--hops 3 \
--max-to-crawl 500 \
--max-to-process 500 \
--restrict-domain \
-f json \
-o .diffbot/tmp/crawl_results.json
Crawl only blog posts (by URL pattern):
~/.diffbot/venv/bin/db crawl "https://example.com" \
--url-process-pattern "/blog/" \
--max-to-process 200
Step 3 — manage jobs
List all active jobs:
~/.diffbot/venv/bin/db crawl-list-jobs
Inspect a specific job:
~/.diffbot/venv/bin/db crawl-list-jobs my-job-name
Delete a job:
~/.diffbot/venv/bin/db crawl-delete-job my-job-name
Deleting a job is permanent on Diffbot's side. crawl-delete-job is not pre-approved:
Claude Code asks before every db command, including each deletion. Only run it when the
user has asked to delete that specific job, and name the job when you do.
Understanding crawl output
The crawl command streams events as they happen. In markdown format (default):
# Job: my-crawl-job-1234567890
- [Success] https://example.com/
- [Success] https://example.com/about
- [!] https://example.com/403-page
[!] marks pages that were crawled but failed extraction (not HTTP failures — those are skipped silently).
In JSON format each event has event_type, timestamp, and details. Event types:
job_created— job was created on the server;details.job_namehas the job nameurl_processed— a page was processed;details.urlanddetails.status(Successor error)
Tips
--restrict-domainis on by default — disable it only if you intentionally want to follow external links.- Use
--url-process-patternto focus extraction on specific path segments without narrowing crawling. - Job names are stable identifiers — always set
--job-nameif you want to look up or delete a job later. - For large crawls, write output to a file with
-orather than reading it all into the conversation.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |