Skip to content

mgiovani/ai-arsenal / skills/anything-to-skill

anything-to-skill

Verified

Turns a docs site, local folder, PDF or book, YouTube video, playlist or channel, or a topic into a source-grounded, token-efficient agent skill (hub SKILL.md, sectioned references, INDEX, evals), and can compact it into a lean expert skill. Use for "make a skill from these docs", "turn this channel into a skill". Not for a skill from your own description (use create-skill), improving one (use improve-skill) or finding third-party skills (use find-skills).

FileKindSize
SKILL.mdskill md29.0 KB
assets/templates/SKILL.md.tmplother53 B
assets/templates/evals.json.tmplother41 B
assets/templates/trigger-eval.json.tmplother9 B
evals/evals.jsonconfig6.2 KB
evals/trigger-eval.jsonconfig1.9 KB
references/compaction.mdmarkdown5.8 KB
references/evals.mdmarkdown2.9 KB
references/extending.mdmarkdown2.6 KB
references/generated-skill-layout.mdmarkdown7.1 KB
references/interview.mdmarkdown2.7 KB
references/judge.mdmarkdown3.5 KB
references/sources.mdmarkdown11.1 KB
scripts/a2s.pyexecutablescript520 B
scripts/anything_to_skill/__init__.pyscript0 B
scripts/anything_to_skill/cli.pyscript17.5 KB
scripts/anything_to_skill/compact/__init__.pyscript0 B
scripts/anything_to_skill/compact/brief.pyscript8.2 KB
scripts/anything_to_skill/compact/freeze.pyscript3.5 KB
scripts/anything_to_skill/compact/gate.pyscript2.8 KB
scripts/anything_to_skill/compact/layout.pyscript1.6 KB
scripts/anything_to_skill/compact/swap.pyscript3.8 KB
scripts/anything_to_skill/compact/verify.pyscript15.4 KB
scripts/anything_to_skill/core/__init__.pyscript0 B
scripts/anything_to_skill/core/models.pyscript3.1 KB
scripts/anything_to_skill/core/sanitize.pyscript2.1 KB
scripts/anything_to_skill/core/ssrf.pyscript1.8 KB
scripts/anything_to_skill/core/store.pyscript13.8 KB
scripts/anything_to_skill/core/tokens.pyscript147 B
scripts/anything_to_skill/core/workspace.pyscript4.7 KB
scripts/anything_to_skill/emit/__init__.pyscript0 B
scripts/anything_to_skill/emit/link.pyscript612 B
scripts/anything_to_skill/emit/templates.pyscript398 B
scripts/anything_to_skill/emit/verify.pyscript9.7 KB
scripts/anything_to_skill/emit/write.pyscript21.0 KB
scripts/anything_to_skill/laya/__init__.pyscript0 B
scripts/anything_to_skill/laya/apply.pyscript3.1 KB
scripts/anything_to_skill/laya/brain.pyscript6.9 KB
scripts/anything_to_skill/laya/client.pyscript2.2 KB
scripts/anything_to_skill/laya/factcheck.pyscript10.5 KB
scripts/anything_to_skill/laya/judge.pyscript6.8 KB
scripts/anything_to_skill/laya/questions.pyscript2.1 KB
scripts/anything_to_skill/plan/__init__.pyscript0 B
scripts/anything_to_skill/plan/brief.pyscript8.1 KB
scripts/anything_to_skill/plan/index.pyscript912 B
scripts/anything_to_skill/plan/pack.pyscript16.6 KB
scripts/anything_to_skill/plan/priority.pyscript5.0 KB
scripts/anything_to_skill/plan/tree.pyscript19.1 KB
scripts/anything_to_skill/scan/__init__.pyscript0 B
scripts/anything_to_skill/scan/credentials.pyscript1.1 KB
scripts/anything_to_skill/scan/injection.pyscript1.9 KB
scripts/anything_to_skill/scan/quotes.pyscript8.4 KB
scripts/anything_to_skill/scan/unicode.pyscript691 B
scripts/anything_to_skill/sources/__init__.pyscript1.2 KB
scripts/anything_to_skill/sources/local/__init__.pyscript0 B
scripts/anything_to_skill/sources/local/convert.pyscript4.6 KB
scripts/anything_to_skill/sources/local/detect.pyscript178 B
scripts/anything_to_skill/sources/local/run.pyscript4.5 KB
scripts/anything_to_skill/sources/local/walk.pyscript1.1 KB
scripts/anything_to_skill/sources/web/__init__.pyscript0 B
scripts/anything_to_skill/sources/web/browser.pyscript4.2 KB
scripts/anything_to_skill/sources/web/detect.pyscript144 B
scripts/anything_to_skill/sources/web/discover.pyscript21.4 KB
scripts/anything_to_skill/sources/web/extract.pyscript13.0 KB
scripts/anything_to_skill/sources/web/fetch.pyscript8.7 KB
scripts/anything_to_skill/sources/web/frontier.pyscript9.5 KB
scripts/anything_to_skill/sources/web/run.pyscript12.6 KB
scripts/anything_to_skill/sources/web/throttle.pyscript7.3 KB
scripts/anything_to_skill/sources/youtube/__init__.pyscript0 B
scripts/anything_to_skill/sources/youtube/asr.pyscript1.5 KB
scripts/anything_to_skill/sources/youtube/captions.pyscript2.4 KB
scripts/anything_to_skill/sources/youtube/detect.pyscript313 B
scripts/anything_to_skill/sources/youtube/frames.pyscript3.8 KB
scripts/anything_to_skill/sources/youtube/funnel.pyscript7.4 KB
scripts/anything_to_skill/sources/youtube/listing.pyscript5.8 KB
scripts/anything_to_skill/sources/youtube/run.pyscript21.5 KB
scripts/anything_to_skill/sources/youtube/vtt.pyscript3.3 KB
scripts/convert_docling.pyexecutablescript513 B
scripts/crawl_brain.pyexecutablescript637 B
scripts/ingest_local.pyexecutablescript463 B
scripts/ingest_web.pyexecutablescript593 B
scripts/ingest_youtube.pyexecutablescript533 B
scripts/judge.pyexecutablescript481 B
scripts/transcribe.pyexecutablescript689 B
scripts/verify_laya.pyexecutablescript925 B
scripts/youtube_brain.pyexecutablescript598 B

scripts/anything_to_skill/sources/web/run.py

View on GitHub
import argparse
import asyncio
import re
import shutil
import time
from http import HTTPStatus
from urllib.parse import urlsplit

from anything_to_skill.core.models import RunSummary, Unit
from anything_to_skill.core.ssrf import SSRFError
from anything_to_skill.core.workspace import Ctx
from anything_to_skill.sources.web import browser
from anything_to_skill.sources.web.discover import (
    boost_linked,
    crawl_scope,
    discover,
    follow_links,
    rerank_pending,
)
from anything_to_skill.sources.web.extract import (
    is_blocked,
    is_js_shell,
    strip_host_boilerplate,
    to_markdown,
)
from anything_to_skill.sources.web.fetch import (
    FetchError,
    FetchResult,
    Net,
    ResponseTooLargeError,
    RobotsDisallowedError,
    close_session,
)
from anything_to_skill.sources.web.frontier import (
    LinkScorer,
    extract_links,
    make_scorer,
)
from anything_to_skill.sources.web.throttle import (
    HostBlockedError,
    HostThrottle,
    RobotsCache,
)

MAX_IDLE_WAIT_S = 60.0
_H1 = re.compile(r'^#\s+(.+)$', re.M)


def add_args(parser: argparse.ArgumentParser) -> None:
    """Add this source's flags; --slug/--base/--max-pages/--allow-private already exist."""
    parser.add_argument(
        '--brain',
        choices=['rules', 'laya'],
        default='rules',
        help='link scorer for the frontier; laya falls back to rules when unavailable',
    )
    parser.add_argument(
        '--discover-only',
        action='store_true',
        help='expand seeds into page units (for estimate) without fetching pages',
    )
    parser.add_argument('--workers', type=int, default=4, help='concurrent fetch tasks')
    parser.add_argument(
        '--min-delay',
        type=float,
        default=1.0,
        help='floor of the per-host delay in seconds (robots Crawl-delay wins when larger)',
    )
    parser.add_argument('--max-depth', type=int, default=4, help='BFS link depth limit')
    parser.add_argument(
        '--max-queue', type=int, default=5000, help='BFS pending-unit cap'
    )
    parser.add_argument(
        '--no-browser',
        action='store_true',
        help='never launch agent-browser; JS shells become needs_js',
    )


class _Crawl:
    def __init__(self, ctx: Ctx, args: argparse.Namespace) -> None:
        self.ctx = ctx
        self.args = args
        self.store = ctx.store
        self.summary = RunSummary()
        throttle = HostThrottle(self.store.log_throttle, floor=args.min_delay)
        self.net = Net(throttle, RobotsCache(), ctx.allow_private)
        self.dropped = ctx.dropped_ids()
        self.dropped_web = sum(
            u.source == 'web' and u.id in self.dropped
            for u in self.store.units(status='done')
        )
        self.claimed = 0
        self.busy = 0
        self.browser_lock = asyncio.Lock()
        self._scorers: dict[str, LinkScorer] = {}

    async def go(self) -> RunSummary:
        try:
            await self.expand_seeds()
            rerank_pending(self.ctx, self.args.brain)
            if not self.args.discover_only:
                if not self.args.no_browser and shutil.which(browser.BIN):
                    self.store.requeue('needs_js')
                await asyncio.gather(*(self.worker() for _ in range(self.args.workers)))
                strip_host_boilerplate(self.store)
        finally:
            await close_session()
        return self.summary

    def error(self, message: str) -> None:
        self.ctx.log(message)
        self.summary.note(message)

    async def expand_seeds(self) -> None:
        for seed in self.store.units(source='web', kind='seed', status='pending'):
            try:
                added = await discover(
                    self.ctx,
                    seed.uri,
                    seed.meta.get('scope'),
                    self.net,
                    self.args.brain,
                )
            except SSRFError as exc:
                self.finish_status(seed, 'skipped', f'ssrf: {exc}')
            except (FetchError, HostBlockedError) as exc:
                self.error(f'seed {seed.uri}: {exc}')
            else:
                self.ctx.log(f'seed {seed.uri}: {added} pages queued')

    def spent(self) -> int:
        return self.store.token_total(exclude=self.dropped)

    def stop(self) -> bool:
        limit = self.ctx.max_pages
        budget = self.ctx.token_budget
        return (limit is not None and self.claimed >= limit) or (
            budget is not None and self.spent() >= budget
        )

    def fits_budget(self) -> bool:
        """False when the next page, plus those in flight, would likely pass the token budget
        (each judged at the largest size so far: a small first page such as a landing page
        would otherwise let a whole wave of big ones through). With no page done yet there is
        no estimate, so a single probe page runs alone first."""
        budget = self.ctx.token_budget
        if budget is None:
            return True
        done = self.store.counts().get('web', {}).get('done', 0) - self.dropped_web
        if not done:
            return not self.busy
        largest = self.store.token_max('web', self.dropped)
        return self.spent() + (self.busy + 1) * largest <= budget

    async def worker(self) -> None:
        while not self.stop():
            if not self.fits_budget():
                if not self.busy:
                    return
                await asyncio.sleep(0.1)
                continue
            claimed = self.store.claim('web')
            if not claimed:
                due = self.store.next_due('web')
                wait = None if due is None else due - time.time()
                if self.busy == 0 and (wait is None or wait > MAX_IDLE_WAIT_S):
                    return
                await asyncio.sleep(
                    0.1 if self.busy else min(max(wait or 0.0, 0.05), 1.0)
                )
                continue
            self.claimed += 1
            self.busy += 1
            try:
                await self.process(claimed[0])
            except Exception as exc:  # noqa: BLE001 - a poison page must not abort the crawl
                self.retry(claimed[0], f'{type(exc).__name__}: {exc}')
            finally:
                self.busy -= 1

    def finish_status(self, unit: Unit, status: str, reason: str) -> None:
        self.store.mark(unit.id, status, reason)
        if hasattr(self.summary, status):
            setattr(self.summary, status, getattr(self.summary, status) + 1)

    def retry(self, unit: Unit, reason: str) -> None:
        if self.store.fail(unit.id, reason) == 'failed':
            self.summary.failed += 1
            self.error(f'{unit.uri}: {reason}')

    async def process(self, unit: Unit) -> None:
        referer = self.store.get(unit.parent).uri if unit.parent else None
        try:
            res = await self.net.get(unit.uri, unit.etag, unit.last_modified, referer)
        except SSRFError as exc:
            self.finish_status(unit, 'skipped', f'ssrf: {exc}')
        except RobotsDisallowedError:
            self.finish_status(unit, 'skipped', 'disallowed by robots.txt')
        except HostBlockedError:
            self.finish_status(unit, 'blocked', 'host circuit breaker open')
        except ResponseTooLargeError as exc:
            self.finish_status(unit, 'skipped', str(exc))
        except FetchError as exc:
            self.retry(unit, str(exc))
        else:
            await self.handle(unit, res)

    async def handle(self, unit: Unit, res: FetchResult) -> None:
        status = res.status
        if status == HTTPStatus.NOT_MODIFIED:
            self.store.not_modified(unit.id)
            self.summary.done += 1
        elif status in (404, 410):
            self.finish_status(unit, 'skipped', f'HTTP {status}')
        elif status == HTTPStatus.FORBIDDEN:
            await self.rendered(unit, res, f'HTTP {status}', 'blocked')
        elif status in (401, 451):
            self.finish_status(unit, 'blocked', f'HTTP {status}')
        elif status == HTTPStatus.OK:
            await self.content(unit, res)
        elif (
            status == HTTPStatus.TOO_MANY_REQUESTS
            or status >= HTTPStatus.INTERNAL_SERVER_ERROR
        ):
            self.retry(unit, f'HTTP {status}')
        else:
            self.finish_status(unit, 'skipped', f'HTTP {status}')

    async def content(self, unit: Unit, res: FetchResult) -> None:
        ctype = res.content_type.lower()
        if 'markdown' in ctype or ctype.startswith('text/plain'):
            m = _H1.search(res.body)
            self.finish(unit, res, res.body, m.group(1).strip() if m else None)
        elif 'html' in ctype or not ctype:
            if is_blocked(res.body):
                await self.rendered(unit, res, 'challenge page', 'blocked')
            elif is_js_shell(res.body):
                await self.rendered(unit, res, 'javascript shell', 'needs_js')
            else:
                md, title = to_markdown(res.body, res.url)
                if not md:
                    self.finish_status(unit, 'skipped', 'no extractable content')
                    return
                self.follow(unit, res, title)
                self.finish(unit, res, md, title)
        else:
            self.finish_status(unit, 'skipped', f'unsupported content-type {ctype}')

    async def rendered(
        self, unit: Unit, res: FetchResult, reason: str, unavailable: str
    ) -> None:
        """agent-browser for a JS shell, a 403 or a challenge page; `unavailable` is the status
        when the browser cannot or may not produce the page.

        401 and 451 never come here, and a challenge the browser cannot pass stays blocked:
        we do not solve CAPTCHAs, logins or paywalls.
        """
        host = urlsplit(res.url).netloc
        state = self.net.throttle.host(host)
        if self.args.no_browser or state.tripped or state.dead:
            return self.finish_status(unit, unavailable, reason)
        async with self.browser_lock:
            try:
                await self.net.throttle.acquire(host)
            except HostBlockedError:  # went dead while this worker waited on the lock
                return self.finish_status(unit, unavailable, reason)
            rendered = None
            try:
                rendered = await asyncio.to_thread(browser.render, res.url)
            except browser.BrowserBlockedError:
                return self.finish_status(
                    unit, 'blocked', f'{reason}; blocked in browser'
                )
            finally:
                self.net.throttle.release(host, 200 if rendered else 0)
        if rendered is None or not await self.browsable(rendered[1]):
            return self.finish_status(unit, unavailable, reason)
        self.finish(unit, res, rendered[0], None)
        return None

    async def browsable(self, final_url: str) -> bool:
        """The page the browser ended on passes the same SSRF and robots gates as any fetch."""
        try:
            await self.net.check(final_url)
        except SSRFError:
            return False
        origin = f'{urlsplit(final_url).scheme}://{urlsplit(final_url).netloc}'
        return self.net.robots.allowed(origin, final_url)

    def finish(
        self, unit: Unit, res: FetchResult, markdown: str, title: str | None
    ) -> None:
        self.store.finish(
            unit.id,
            markdown,
            title=title,
            hint={'final_url': res.url} if res.url != unit.uri else None,
            meta={'content_type': res.content_type.split(';')[0].strip()},
            etag=res.etag,
            last_modified=res.last_modified,
        )
        self.summary.done += 1

    def scorer(self, scope: str) -> LinkScorer:
        if scope not in self._scorers:
            self._scorers[scope] = make_scorer(
                self.ctx.goal, self.args.brain, scope, self.ctx.log
            )
        return self._scorers[scope]

    def follow(self, unit: Unit, res: FetchResult, title: str | None) -> None:
        """Feed the BFS frontier, or promote the listed pages this page links to."""
        found = crawl_scope(self.store, unit.uri)
        if found is None:
            return
        scope, mode = found
        links = extract_links(res.body, res.url)
        if mode == 'bfs':
            follow_links(
                self.ctx,
                unit.id,
                unit.depth,
                links,
                self.scorer(scope),
                scope,
                title or '',
                max_depth=self.args.max_depth,
                max_queue=self.args.max_queue,
            )
        else:
            boost_linked(self.ctx, unit, title or '', links, scope)


def run(ctx: Ctx, args: argparse.Namespace) -> RunSummary:
    return asyncio.run(_Crawl(ctx, args).go())
canvas-designanthropics/skillsCreate beautiful visual art in .png and .pdf documents using design philosophy. You should use this skill when the user asks to create a poster, piece of art, design, or other static piece. Create original visual designs, never copying existing artists' work to avoid copyright violations.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
docxanthropics/skillsUse this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files) or Word templates (.dotx files). Triggers include: any mention of 'Word doc', 'word document', '.docx', '.dotx', or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting or reorganizing content from .docx or .dotx files, inserting or replacing images in documents, performing find-and-replace in Word files, working with tracked changes or comments, or converting content into a polished Word document. If the user asks for a 'report', 'memo', 'letter', 'template', or similar deliverable as a Word or .docx file, use this skill. Do NOT use for PDFs, spreadsheets, Google Docs, or general coding tasks unrelated to document generation.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
pdfanthropics/skillsUse this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
pptxanthropics/skillsUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an email or summary); editing, modifying, or updating existing presentations; combining or splitting slide files; working with templates (.potx), layouts, speaker notes, or comments. Trigger whenever the user mentions "deck," "slides," "presentation," or references a .pptx or .potx filename, regardless of what they plan to do with the content afterward. If a .pptx or .potx file needs to be opened, created, or touched, use this skill.Stars: 198kUpdated Claude CodeCodexVerifiedInstall