Skip to content

yya007/SkillFinder / skills/crawl-sources

crawl-sources

Verified

Crawl all SkillFinder data sources (SkillsMP, ClawHub, SkillHub, Anthropic marketplace, GitHub topics) and write fresh raw JSONL files to data/raw/. Runs crawlers in parallel; supports incremental resume and per-source limits.

crawl-sources

Crawl all skill registries and refresh the raw data files in data/raw/.

This skill covers the crawl step only — it does not normalize, embed, or rebuild the index. Run update-index afterward to turn fresh raw data into a searchable index.

Prerequisites

  • GITHUB_TOKEN env var set (required by SkillsMP, ClawHub, and topic crawlers)
  • Python dependencies installed: pip install -r requirements-dev.txt

Agent Instructions

When this skill triggers, determine scope from the user's request:

User saysScope
"crawl everything" / "all sources"Run all five crawlers (default)
"crawl skillsmp" / "crawl github"SkillsMP only
"crawl clawhub" / "crawl openclaw"ClawHub only
"crawl skillhub"SkillHub only
"crawl marketplace" / "crawl anthropic"Marketplace only
"crawl topics" / "crawl github topics"Topic crawler only
"quick test" / "limit N"Add --limit N to each crawler

Step 1 — Check prerequisites

echo "${GITHUB_TOKEN:0:4}..."

If empty, stop:

"Set GITHUB_TOKEN before crawling: export GITHUB_TOKEN=ghp_..."

mkdir -p data/raw

Step 2 — Run crawlers

All sources (parallel — default):

python -m crawlers.skillsmp_crawler -o data/raw/skillsmp.jsonl &
python -m crawlers.clawhub_crawler  -o data/raw/clawhub.jsonl &
python -m crawlers.skillhub_crawler -o data/raw/skillhub.jsonl &
python -m crawlers.marketplace_crawler -o data/raw/marketplace.jsonl &
python -m crawlers.topic_crawler    -o data/raw/topic.jsonl --data-dir data/raw &
wait

Single-source variants (use when user targets one source):

# SkillsMP (GitHub code search for SKILL.md files; ~30 min full run)
python -m crawlers.skillsmp_crawler -o data/raw/skillsmp.jsonl

# ClawHub / OpenClaw (awesome-list + org/topic discovery)
python -m crawlers.clawhub_crawler -o data/raw/clawhub.jsonl

# SkillHub (HTML scrape with pagination)
python -m crawlers.skillhub_crawler -o data/raw/skillhub.jsonl

# Anthropic official marketplace
python -m crawlers.marketplace_crawler -o data/raw/marketplace.jsonl

# GitHub topic tags (claude-skill, codex-skill, agent-skill, …)
python -m crawlers.topic_crawler -o data/raw/topic.jsonl --data-dir data/raw

Useful flags (apply to any crawler):

FlagEffect
--limit NCap at N records — use for quick tests
--mode incrementalSkip repos already present in the output file (preferred over --resume)
--mode fullComplete re-crawl (default)
--mode metadataRefresh stars/ETags only, skip content fetch
--mode discoverOnly fetch repos pushed since last run (date-filtered search)
--resumeDeprecated — use --mode incremental instead
--since YYYY-MM-DDOnly include repos pushed after this date (SkillsMP)
--log-level DEBUGVerbose output
--filter-cache FILEPath to dedup cache (default: data/filter_cache.jsonl)

Orchestrator (runs all crawlers in the right order):

python pipeline/update_crawl.py --mode incremental
python pipeline/update_crawl.py --mode full --sources clawhub,skillsmp
python pipeline/update_crawl.py --mode incremental --chain  # also runs normalize/embed/build

Step 3 — Report record counts

After all crawlers finish:

for f in data/raw/skillsmp.jsonl data/raw/clawhub.jsonl data/raw/skillhub.jsonl data/raw/marketplace.jsonl data/raw/topic.jsonl; do
  [ -f "$f" ] && echo "$(wc -l < $f) $(basename $f)"
done

Report the counts to the user. Flag any source with 0 records as a potential failure.


Step 4 — Suggest next step

If crawl succeeded, offer:

"Raw data is refreshed. Run the update-index skill to normalize, embed, and rebuild the FAISS index — or run it now?"

If the user says yes, hand off to the update-index skill starting at its Step 3 (Backfill metadata).


Troubleshooting

SymptomLikely causeFix
SkillsMP returns 0 recordsRate limit or bad tokenCheck GITHUB_TOKEN; retry with --limit 50
ClawHub hangsGitHub API slowAdd --limit 200 for a quick run
SkillHub returns few recordsSite structure changedRun with --log-level DEBUG and inspect HTML
filter_cache.jsonl grows largeNormal — dedup cacheSafe to delete; will be rebuilt on next run
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 469kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 321kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 321kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 321kUpdated Claude CodeCodexVerifiedInstall