Skip to content

diffbot/diffbot-skills / skills/diffbot-crawl

diffbot-crawl

Verified

Crawls a website from a seed URL and extracts every page it finds into markdown or typed JSON. Prefer when a task needs to discover a catalog of pages from an entire site, or deep search a site for structured data (e.g. all product inventory, all press releases, PDFs). Crawls run on Diffbot's servers; the CLI streams each page as it is processed and can list or delete jobs. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl, every page on, all pages of, whole site.

FileKindSize
SKILL.mdskill md5.5 KB
---
name: diffbot-crawl
description: "Crawls a website from a seed URL and extracts every page it finds into markdown or typed JSON. Prefer when a task needs to discover a catalog of pages from an entire site, or deep search a site for structured data (e.g. all product inventory, all press releases, PDFs). Crawls run on Diffbot's servers; the CLI streams each page as it is processed and can list or delete jobs. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl, every page on, all pages of, whole site."
---

# Diffbot Crawler

Crawl websites for structured content via the Diffbot Crawlbot API, and manage crawler jobs.

## The `db crawl*` CLI commands

**Always invoke with the fixed path `~/.diffbot/venv/bin/db`.**

```
~/.diffbot/venv/bin/db crawl <SITE> [options]            # start a crawl and stream events
~/.diffbot/venv/bin/db crawl-list-jobs [JOB_NAME]        # list all jobs or inspect one
~/.diffbot/venv/bin/db crawl-delete-job <JOB_NAME>       # delete a job
```

### `db crawl` options

| Flag | Default | Description |
|------|---------|-------------|
| `--hops` | `2` | Maximum link depth from seed URLs |
| `--job-name` | auto-generated | Name for the crawler job |
| `--max-to-crawl` | `100` | Maximum pages to crawl |
| `--max-to-process` | `100` | Maximum pages to process |
| `--restrict-domain` | `true` | Only follow links on the same domain as seeds |
| `--api-url` | `""` | Diffbot API endpoint for processing (default: auto/analyze) |
| `--crawl-delay` | `-1` | Delay between requests to same domain (seconds) |
| `--url-crawl-pattern` | none | Only crawl URLs matching this pattern |
| `--url-process-pattern` | none | Only process URLs matching this pattern |
| `--obey-robots` | off | Obey robots.txt |
| `--use-proxies` | off | Use proxies for crawling |
| `--custom-headers` | none | Newline-separated custom HTTP headers |
| `-f`, `--format` | `markdown` | Output: `markdown` or `json` |
| `-o`, `--output` | stdout | Write output to file |

## Workflow

### Step 1 — check the CLI

```
~/.diffbot/venv/bin/db --version
```

The `db` CLI is a one-time **user** install at `~/.diffbot/venv/bin/db`, the same way `gh` or
`wrangler` is — **this skill never installs, upgrades, or uninstalls it.** If `--version`
fails with `No such file or directory`, stop and show the user this line to run or approve
themselves (it is deliberately not pre-approved):

```
python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install 'diffbot>=3.4.0,<4'
```

If it prints a version below 3.4.0 (including 3.1.x and 3.2.x — upgrade them), fails with `No such option '--version'` (an old
`diffbot-python` install, which predates the flag), or warns about the `diffbot-python`
rename, point the user at **Upgrading the `db` CLI** in the plugin README and stop. Never run an install,
upgrade, or uninstall line yourself. 3.1.0 added `--version` and the rename; 3.2.0 switched DQL to POST; 3.4.0 is the release these skills require.

Save output files under `.diffbot/tmp/` in the working directory. `db` creates that folder itself, so don't `mkdir`
it first. `db`, `jq`, and `cat` are not pre-approved — Claude will ask before running them. Always tell the user where an export was saved.

The token is read from `DIFFBOT_API_TOKEN` env var or `~/.diffbot/credentials`. If missing:

```
echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials
```

### Step 2 — crawl a site

Basic crawl with defaults (100 pages, 2 hops deep):

```
~/.diffbot/venv/bin/db crawl "https://example.com"
```

Deep crawl of a documentation site, saving to JSON:

```
~/.diffbot/venv/bin/db crawl "https://docs.example.com" \
  --hops 3 \
  --max-to-crawl 500 \
  --max-to-process 500 \
  --restrict-domain \
  -f json \
  -o .diffbot/tmp/crawl_results.json
```

Crawl only blog posts (by URL pattern):

```
~/.diffbot/venv/bin/db crawl "https://example.com" \
  --url-process-pattern "/blog/" \
  --max-to-process 200
```

### Step 3 — manage jobs

List all active jobs:

```
~/.diffbot/venv/bin/db crawl-list-jobs
```

Inspect a specific job:

```
~/.diffbot/venv/bin/db crawl-list-jobs my-job-name
```

Delete a job:

```
~/.diffbot/venv/bin/db crawl-delete-job my-job-name
```

Deleting a job is permanent on Diffbot's side. `crawl-delete-job` is not pre-approved:
Claude Code asks before every `db` command, including each deletion. Only run it when the
user has asked to delete that specific job, and name the job when you do.

## Understanding crawl output

The `crawl` command streams events as they happen. In markdown format (default):

```
# Job: my-crawl-job-1234567890

- [Success] https://example.com/
- [Success] https://example.com/about
- [!] https://example.com/403-page
```

`[!]` marks pages that were crawled but failed extraction (not HTTP failures — those are skipped silently).

In JSON format each event has `event_type`, `timestamp`, and `details`. Event types:
- `job_created` — job was created on the server; `details.job_name` has the job name
- `url_processed` — a page was processed; `details.url` and `details.status` (`Success` or error)

## Tips

- `--restrict-domain` is on by default — disable it only if you intentionally want to follow external links.
- Use `--url-process-pattern` to focus extraction on specific path segments without narrowing crawling.
- Job names are stable identifiers — always set `--job-name` if you want to look up or delete a job later.
- For large crawls, write output to a file with `-o` rather than reading it all into the conversation.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall