Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/browser-tools

browser-tools

Verified


name: browser-tools description: Router for all browser automation + web scraping + agent-on-the-web tools in Cianfhoghlaim. Use this to decide which tool fits a task: Crawl4AI (self-hosted REST), Crawl4AI MCP (v0.9.x native), Firecrawl MCP, Firecrawl CLI, Skyvern, Stagehand, or Playwright/CDP. Covers when to use each, the auth + cookie patterns, the 6-backend architecture (3 default + 2 opt-in + 1 MCP-native), the new Crawl4AI v0.9.x features (native MCP server, secure-by-default, JWT auth), the opt-in pattern for Skyvern + Stagehand, and the Cianfhoghlaim safety rules (domain allowlist, no unscraped authentication flows). Triggers: 'browse website', 'scrape URL', 'login flow', 'click button', 'extract data from page', 'browser agent', 'autonomous browsing', 'screenshot', 'PDF capture', 'authenticated scraping', 'Crawl4AI', 'deep crawl', 'MCP-native'.

Browser Tools — Router (post v4 + Crawl4AI v0.9.x + native MCP)

Cianfhoghlaim has 6 ways to drive a browser, scrape a page, or run an agent-on-the-web. This skill is the router — pick the right one for the task.

The 6 backends (Moderate strategy, 6 = 4 default + 2 opt-in)

BackendDefault?CostPortWhen to use
Crawl4AI REST (self-hosted)✅ ON$011235The default Python SDK path. CSS+LLM extraction, deep crawl, hooks. Best for Python pipelines.
Crawl4AI MCP (self-hosted, v0.9.x native)✅ ON$011235NEW 2026-08-21: the native MCP server (/mcp/sse, /mcp/ws). JWT-authed. Best for MCP-native agents.
Firecrawl MCP (paid)✅ ON$0.005-0.05/pageMCPPaid fallback. Best for anti-bot + JS-rendered + agent/research.
Playwright CDP (self-hosted)✅ ON$09222Drive a real browser locally. Best for fine-grained interactions.
Skyvern (opt-in)⚙️ BROWSER_ENABLE_SKYVERN=1$08000Vision-based semantic navigation. Opt-in for vision-heavy flows.
Stagehand (opt-in)⚙️ BROWSER_ENABLE_STAGEHAND=1$03100AI-powered UI interactions + agent mode. Opt-in for autonomous flows.

Browserbase was removed 2026-06-29 (no credits, no replacement plan) and re-confirmed 2026-08-21 (per openspec/changes/2026-08-21-complete-browserbase-archive-and-crawl4ai-mcp-v1/). The 2 opt-in backends are OFF by default to keep the surface area small.

The 5 corresponding skills

If you need to…Use this toolSkill
Bulk extraction from a single URL into markdown/JSON (Python pipeline)Crawl4AI RESTcrawl4ai
Bulk extraction from many URLs (deep crawl)Crawl4AI REST (BFS/DFS)crawl4ai
MCP-native bulk extraction (MCP client / OpenCode agent)Crawl4AI MCP (v0.9.x)n/a — mcp:smoke:crawl4ai
Agent-driven research + structured extractionFirecrawl (MCP) or CLIfirecrawl / firecrawl-cli
Cloud-based anti-bot + JS-rendered scrapingFirecrawl (MCP)firecrawl
Local Chromium / Firefox via PlaywrightPlaywright (via the browser stack)browser
Vision-based semantic navigationSkyvern (opt-in)n/a — set BROWSER_ENABLE_SKYVERN=1
AI-powered UI interactions + agent modeStagehand (opt-in)n/a — set BROWSER_ENABLE_STAGEHAND=1

Decision tree (Crawl4AI-first)

Need to scrape a single static page (Python pipeline)?  → Crawl4AI REST (use css strategy)
Need MCP-native bulk extraction (agent runtime)?        → Crawl4AI MCP (v0.9.x native)
Need to extract structured data with a schema?        → Crawl4AI (use LLM strategy)
Need to deep-crawl a docs site?                        → Crawl4AI (BFS/DFS strategy)
Need to click / login / fill a form?                   → Firecrawl MCP interact (or Playwright if local)
Need vision-based navigation?                          → Skyvern (opt-in)
Need an autonomous agent that can plan?                 → Stagehand agent mode (opt-in)
Need a self-hosted scraper with no LLM cost?            → Crawl4AI (use css strategy — free)
Need full Playwright control locally?                   → browser (Playwright stack)
Need to monitor a page for changes?                     → Firecrawl MCP monitor
Need to interact with a page (login + click)?           → Firecrawl MCP interact
Need to scrape many URLs in batch?                      → Crawl4AI batch or Firecrawl batch

The 5 backends in Python

from cianfhoghlaim.core.browser import BrowserClient, BackendType

client = BrowserClient()

# 1. Crawl4AI (default, self-hosted, port 11235)
result = client.search(backend=BackendType.CRAWL4AI_LOCAL, query="...")

# 2. Firecrawl (default, paid MCP)
result = client.search(backend=BackendType.FIRECRAWL_MCP, query="...")

# 3. Playwright CDP (default, self-hosted, port 9222)
result = client.search(backend=BackendType.CDP_LOCAL, query="...")

# 4. Skyvern (opt-in, only if BROWSER_ENABLE_SKYVERN=1)
result = client.search(backend=BackendType.SKYVERN_LOCAL, query="...")

# 5. Stagehand (opt-in, only if BROWSER_ENABLE_STAGEHAND=1)
result = client.search(backend=BackendType.STAGEHAND_LOCAL, query="...")

The new Crawl4AI 0.7.4 features (Phase E)

FeatureUse caseAPI
JsonCssExtractionStrategyZero-cost extraction for known-structure pages (NCCA, SEC, DES)client.extract_with_css(url, schema)
LLMExtractionStrategyType-safe structured extraction with Pydanticclient.extract_with_llm(url, PydanticClass)
use_managed_browser=TruePersistent login sessions (QubStudent, etc.)client.authenticate(profile_name)
BFSDeepCrawlStrategy / DFSDeepCrawlStrategyFull-site crawlingclient.bulk_crawl(seed_url, strategy="BFS")
Hooks (on_page_context_created, etc.)Login automation + cookie capturen/a (advanced)

Cianfhoghlaim safety rules

  1. No unscraped authentication flows — any login / OAuth / MFA interaction MUST go through Firecrawl interact (which can use a persistent context) or a manually-set cookie. Never inline credentials into a script.
  2. Domain allowlist for autonomous agents — any agent that browses without human supervision must use the new opt-in Stagehand (with an explicit allowlist) or the Skyvern vision-based navigation. Unbounded browse is a credit-burn hazard.
  3. No raw :latest for upstream images — the browser stack uses pinned images (Crawl4AI 0.7.4, Skyvern 1.x.y, Stagehand 1.x.y).
  4. Token storage — Firecrawl + Skyvern + Stagehand tokens come from the Infisical dev-baile vault. Never commit them to .env or to the source tree.
  5. Credit budget guard — every Firecrawl call is routed through CreditBudget (SQLite-backed, persistent across restarts). When the budget is depleted, the router falls back to a free backend (Crawl4AI). The budget never raises BudgetExhaustedError to the caller.

When to use Firecrawl MCP vs CLI

Use MCP (firecrawl/SKILL.md) when the agent runtime has the Firecrawl MCP server configured (the Cianfhoghlaim default; the MCP tools are visible in the agent's tool palette).

Use CLI (firecrawl-cli/SKILL.md) when:

  • Running in CI / scheduled jobs (no agent runtime).
  • Needing a specific endpoint the MCP doesn't expose (e.g. firecrawl monitor, firecrawl browser-sandbox).
  • Working from a terminal where you want the human-readable progress output.

The two are functionally equivalent for the 4 core endpoints (scrape, crawl, search, extract); MCP just wraps them as MCP tools.

When to use the opt-in backends

The 5-backend Moderate strategy keeps Skyvern + Stagehand OFF by default to minimize surface area. To enable them, set the env vars:

# In .env or mise.toml
BROWSER_ENABLE_SKYVERN=1
BROWSER_ENABLE_STAGEHAND=1

When enabled, the corresponding DLT sources + Dagster assets are registered (per the defs/browser/auth_assets.py pattern).

Cross-references

  • The canonical Python module: cianfhoghlaim.core.browser (formerly sruth_browser; renamed 2026-06-29 per openspec/changes/2026-06-29-browser-stack-crawl4ai-refactor)
  • The browser stack YAML files: bonneagar/stacks/browser/ (v4 worktree split; bug fixes need 2 PRs)
  • The new Dagster Component: cianfhoghlaim/assets/_cianfhoghlaim_dagster_defs/defs/browser/ (DltLoadCollectionComponent + crawl4ai_defs + firecrawl_defs + auth_defs)
  • The BAML extraction: cianfhoghlaim/core/baml/_cianfhoghlaim_src/author_archive.baml (calls ScrapeStrategist for site map + bulk scrape)
  • The DAG asset groups that use this: author_archive_assets.py, leabharlann_inbox_assets.py, croilar_cv_extraction.py, official_media/scraping_assets.py

History

  • 2026-08-14: Firecrawl MCP promoted to the canonical external search surface of the agent stack (per the 2026-08-14-firecrawl-mcp-ccc-dual-search-v1 change). The FirecrawlMCPClient wrapper at agents/meaisinfhoghlaim/firecrawl_mcp/client.py exposes all 12 MCP tools with Pydantic + Langfuse @observe. The complete routing table (ccc vs cognee vs firecrawl_mcp) lives at AGENTS.md §"Triple-search architecture" and the dual-search-architecture spec.
  • 2026-06-28: v4 consolidation (5 quadrants → 1 cianfhoghlaim package)
  • 2026-06-29: Browser stack + Crawl4AI refactor
    • Phase A: DAG integration (new defs/browser/ Component)
    • Phase B: killed browserbase (no credits)
    • Phase D: cut to 3 backends, added Skyvern + Stagehand back as opt-in
    • Phase E: new Crawl4AI 0.7.4 features (CSS+LLM, auth, deep crawl)
    • Phase F: renamed sruth_browser → browser (v4 cleanup)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall