Skip to content

diffbot/diffbot-skills / skills/diffbot-crawl

diffbot-crawl

Verified

Crawls a website from a seed URL and extracts every page it finds into markdown or typed JSON. Prefer when a task needs to discover a catalog of pages from an entire site, or deep search a site for structured data (e.g. all product inventory, all press releases, PDFs). Crawls run on Diffbot's servers; the CLI streams each page as it is processed and can list or delete jobs. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl, every page on, all pages of, whole site.

Diffbot Crawler

Crawl websites for structured content via the Diffbot Crawlbot API, and manage crawler jobs.

The db crawl* CLI commands

Always invoke with the fixed path ~/.diffbot/venv/bin/db.

~/.diffbot/venv/bin/db crawl <SITE> [options]            # start a crawl and stream events
~/.diffbot/venv/bin/db crawl-list-jobs [JOB_NAME]        # list all jobs or inspect one
~/.diffbot/venv/bin/db crawl-delete-job <JOB_NAME>       # delete a job

db crawl options

FlagDefaultDescription
--hops2Maximum link depth from seed URLs
--job-nameauto-generatedName for the crawler job
--max-to-crawl100Maximum pages to crawl
--max-to-process100Maximum pages to process
--restrict-domaintrueOnly follow links on the same domain as seeds
--api-url""Diffbot API endpoint for processing (default: auto/analyze)
--crawl-delay-1Delay between requests to same domain (seconds)
--url-crawl-patternnoneOnly crawl URLs matching this pattern
--url-process-patternnoneOnly process URLs matching this pattern
--obey-robotsoffObey robots.txt
--use-proxiesoffUse proxies for crawling
--custom-headersnoneNewline-separated custom HTTP headers
-f, --formatmarkdownOutput: markdown or json
-o, --outputstdoutWrite output to file

Workflow

Step 1 — check the CLI

~/.diffbot/venv/bin/db --version

The db CLI is a one-time user install at ~/.diffbot/venv/bin/db, the same way gh or wrangler is — this skill never installs, upgrades, or uninstalls it. If --version fails with No such file or directory, stop and show the user this line to run or approve themselves (it is deliberately not pre-approved):

python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install 'diffbot>=3.4.0,<4'

If it prints a version below 3.4.0 (including 3.1.x and 3.2.x — upgrade them), fails with No such option '--version' (an old diffbot-python install, which predates the flag), or warns about the diffbot-python rename, point the user at Upgrading the db CLI in the plugin README and stop. Never run an install, upgrade, or uninstall line yourself. 3.1.0 added --version and the rename; 3.2.0 switched DQL to POST; 3.4.0 is the release these skills require.

Save output files under .diffbot/tmp/ in the working directory. db creates that folder itself, so don't mkdir it first. db, jq, and cat are not pre-approved — Claude will ask before running them. Always tell the user where an export was saved.

The token is read from DIFFBOT_API_TOKEN env var or ~/.diffbot/credentials. If missing:

echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials

Step 2 — crawl a site

Basic crawl with defaults (100 pages, 2 hops deep):

~/.diffbot/venv/bin/db crawl "https://example.com"

Deep crawl of a documentation site, saving to JSON:

~/.diffbot/venv/bin/db crawl "https://docs.example.com" \
  --hops 3 \
  --max-to-crawl 500 \
  --max-to-process 500 \
  --restrict-domain \
  -f json \
  -o .diffbot/tmp/crawl_results.json

Crawl only blog posts (by URL pattern):

~/.diffbot/venv/bin/db crawl "https://example.com" \
  --url-process-pattern "/blog/" \
  --max-to-process 200

Step 3 — manage jobs

List all active jobs:

~/.diffbot/venv/bin/db crawl-list-jobs

Inspect a specific job:

~/.diffbot/venv/bin/db crawl-list-jobs my-job-name

Delete a job:

~/.diffbot/venv/bin/db crawl-delete-job my-job-name

Deleting a job is permanent on Diffbot's side. crawl-delete-job is not pre-approved: Claude Code asks before every db command, including each deletion. Only run it when the user has asked to delete that specific job, and name the job when you do.

Understanding crawl output

The crawl command streams events as they happen. In markdown format (default):

# Job: my-crawl-job-1234567890

- [Success] https://example.com/
- [Success] https://example.com/about
- [!] https://example.com/403-page

[!] marks pages that were crawled but failed extraction (not HTTP failures — those are skipped silently).

In JSON format each event has event_type, timestamp, and details. Event types:

  • job_created — job was created on the server; details.job_name has the job name
  • url_processed — a page was processed; details.url and details.status (Success or error)

Tips

  • --restrict-domain is on by default — disable it only if you intentionally want to follow external links.
  • Use --url-process-pattern to focus extraction on specific path segments without narrowing crawling.
  • Job names are stable identifiers — always set --job-name if you want to look up or delete a job later.
  • For large crawls, write output to a file with -o rather than reading it all into the conversation.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall