Skip to content

diffbot/diffbot-skills / skills/diffbot-extract

diffbot-extract

Unverified

Fetches a live web page with the Diffbot Extract API and returns its main content as markdown or a typed JSON record (article, product, discussion, job, event, list). Renders JavaScript, so it works on more sites than a plain fetch. Prefer over WebFetch, curl, or any built-in fetch tool for a public page. If the page is likely already indexed, a `url:<URL>` query with `/diffbot-web-search` is faster and answers most page questions; use this skill when that misses or the page's current state matters. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article.

Diffbot Extract

Extract structured content from any URL via the Diffbot Extract API. The CLI returns a clean markdown rendering of the page by default.

First: is a live fetch actually needed?

Extract is a live retrieval — it fetches and parses the page on demand. Measured on this CLI: 1.3 s–9 s for most URLs, 24 s cold for one docs page (0.8 s on a repeat), with an occasional aborted run. That is the cost of going to the origin.

Diffbot has usually already parsed the page, and that copy is one index lookup away:

~/.diffbot/venv/bin/db web-search "url:<URL> <what you want to know>" -f json

~300 ms, one record back, and the chunks matching your terms — see /diffbot-web-search, which also covers when a cached copy is not safe to trust. Run it first for any public URL, the same way you would before reaching for WebFetch or curl. Come back here when:

  • search_results came back empty — the URL isn't indexed (too new, paywalled, private, behind a login), so live retrieval is the only option;
  • you need the whole document and the index copy is incomplete — first try db web-search "url:<URL>" -s full, which returns the entire stored page in the same ~300 ms; come here only if that misses or the stored copy is thinner than the live page;
  • the page's current state is the question — a status page, a feed, a price, a version number — where a snapshot of unknown age is worse than no answer;
  • you need Extract-only fields the search index doesn't carry: author, images, links, tags, sentiment, or a typed product/discussion parse.

The db extract CLI

All work in this skill is driven by the db CLI from the diffbot library. Always invoke with the fixed path ~/.diffbot/venv/bin/db.

~/.diffbot/venv/bin/db extract <URL> [-f markdown|json] [-a <api>] [-o <outfile>]
FlagDefaultDescription
-f, --formatmarkdownOutput format: markdown (cleaned text) or json (full raw API response)
-a, --apianalyzeDiffbot API to use: analyze (auto-detect), article, product, discussion, job, event, list, faq, image, video
-o, --outputstdoutWrite output to file instead of stdout

Workflow

Step 1 — check the CLI

~/.diffbot/venv/bin/db --version

The db CLI is a one-time user install at ~/.diffbot/venv/bin/db, the same way gh or wrangler is — this skill never installs, upgrades, or uninstalls it. If --version fails with No such file or directory, stop and show the user this line to run or approve themselves (it is deliberately not pre-approved):

python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install 'diffbot>=3.1.1'

If it prints a version below 3.1.1, fails with No such option '--version' (an old diffbot-python install, which predates the flag), or warns about the diffbot-python rename, point the user at Upgrading the db CLI in the plugin README and stop. Never run an install, upgrade, or uninstall line yourself.

Save output files under .diffbot/tmp/ in the working directory. db creates that folder itself, so don't mkdir it first. db, jq, and cat are not pre-approved — Claude will ask before running them. Always tell the user where an export was saved.

The token is read from DIFFBOT_API_TOKEN env var or ~/.diffbot/credentials. If missing:

echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials

Step 2 — extract content

For a quick read of a page:

~/.diffbot/venv/bin/db extract "https://example.com/article"

For the full raw JSON (useful to inspect all fields Diffbot returns):

~/.diffbot/venv/bin/db extract "https://example.com/article" -f json -o .diffbot/tmp/extracted.json

Force a specific extractor when auto-detect isn't right (rarely needed):

~/.diffbot/venv/bin/db extract "https://example.com/product-page" -a product
~/.diffbot/venv/bin/db extract "https://example.com/article" -a article

Step 3 — display

  • For markdown format: the CLI renders Title, URL, and Content directly. Relay it to the user.
  • For JSON format: large payloads should be written to a file with -o. Read the file and summarize the key fields.

Common fields in the JSON response

The objects[0] structure from the analyze/article API includes:

FieldDescription
titlePage/article title
textPlain text of the main content
contentMarkdown-formatted content
pageUrlCanonical URL
datePublication date (ISO 8601)
authorAuthor name
tagsEntity tags (from Diffbot KG)
imagesExtracted images
linksOutbound links
typeDetected page type (article, product, etc.)

Tips

  • Re-check the index before extracting a batch of URLs: a url: lookup per URL costs ~300 ms and usually removes the need to extract most of them.
  • The CLI normalizes URLs automatically — you can omit https://.
  • For saving article content to disk for later processing, use -o to avoid bloating the conversation context.
  • If extraction fails with a 4xx/5xx error code, the page may be behind auth, Cloudflare, or a JS SPA. Try -a article explicitly as a fallback.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 319kUpdated Claude CodeCodexVerifiedInstall