Skip to content

code-yeongyu/oh-my-openagent / packages/shared-skills/skills/ultimate-browsing

ultimate-browsing

Flagged

Reaches web pages the user's browser cannot open: WAF-blocked and bot-scored hosts, platform-native readers (Chinese and social platforms), archive snapshots with provenance, and a browser you own for stealth or throwaway profiles. Not for plain search, an unblocked static fetch, or a page the user's browser can open.

FileKindSize
.gitignoreother29 B
ATTRIBUTION.mdmarkdown2.5 KB
SKILL.mdskill md11.9 KB
engine/AGENTS.mdmarkdown10.8 KB
engine/__init__.pyscript668 B
engine/__main__.pyscript5.0 KB
engine/bias_check.pyscript7.1 KB
engine/curl_probe.pyscript2.4 KB
engine/executor.pyscript8.0 KB
engine/fetch_chain.pyscript13.0 KB
engine/referers.pyscript327 B
engine/result_schema.pyscript1.5 KB
engine/summary.pyscript1.2 KB
engine/surrogate.pyscript7.9 KB
engine/surrogates.yamlconfig2.3 KB
engine/templates/knip.workspace.jsoncother1.5 KB
engine/templates/package.jsonmanifest349 B
engine/templates/playwright_mobile_chrome.jsscript3.6 KB
engine/templates/playwright_real_chrome.jsscript5.5 KB
engine/tests/fixtures/amp_redirect_stub.htmlother323 B
engine/tests/fixtures/search_interstitial.htmlother90.4 KB
engine/tests/fixtures/wayback_available.jsonconfig246 B
engine/tests/fixtures/wayback_snapshot.htmlother181.4 KB
engine/tests/test_executor_timeout.pyscript6.2 KB
engine/tests/test_fetch_chain.pyscript2.7 KB
engine/tests/test_playwright_stealth.pyscript3.9 KB
engine/tests/test_playwright_templates.pyscript10.7 KB
engine/tests/test_surrogate.pyscript11.3 KB
engine/tests/test_surrogate_validators.pyscript3.5 KB
engine/url_transforms.pyscript3.2 KB
engine/validators.pyscript9.1 KB
engine/waf_detector.pyscript6.9 KB
engine/waf_profiles.yamlconfig6.2 KB
references/agent-reach/README.mdmarkdown2.2 KB
references/agent-reach/career.mdmarkdown755 B
references/agent-reach/dev.mdmarkdown1.3 KB
references/agent-reach/search.mdmarkdown821 B
references/agent-reach/social.mdmarkdown5.6 KB
references/agent-reach/video.mdmarkdown2.7 KB
references/agent-reach/web.mdmarkdown2.1 KB
references/chrome-stealth.mdmarkdown2.4 KB
references/insane-search/README.mdmarkdown20.2 KB
references/insane-search/cache-archive.mdmarkdown4.1 KB
references/insane-search/fallback.mdmarkdown6.9 KB
references/insane-search/jina.mdmarkdown4.3 KB
references/insane-search/json-api.mdmarkdown3.5 KB
references/insane-search/media.mdmarkdown3.5 KB
references/insane-search/metadata.mdmarkdown2.9 KB
references/insane-search/naver.mdmarkdown3.4 KB
references/insane-search/playwright.mdmarkdown6.8 KB
references/insane-search/public-api.mdmarkdown3.4 KB
references/insane-search/rss.mdmarkdown3.0 KB
references/insane-search/tls-impersonate.mdmarkdown6.5 KB
references/insane-search/twitter.mdmarkdown3.5 KB
scripts/cookie_crypto.pyscript2.9 KB
scripts/cookie_domains.pyscript1.1 KB
scripts/cookie_paths.pyscript3.8 KB
scripts/extract_cookies.pyscript9.4 KB
scripts/tests/test_cookie_domain_filter.pyscript3.5 KB
scripts/tests/test_extract_cookies.pyscript9.4 KB

engine/url_transforms.py

View on GitHub
"""Generic URL transforms for the fetch grid.

Transforms are domain-agnostic *rules*. They never reference a specific
site by name. A transform either applies (returns a new URL) or is skipped
(returns None). Callers iterate transforms in order.

Empirically useful transforms (see observations/):
  * mobile_subdomain — `www.example.com` → `m.example.com`
    Strong win on SSR sites with mobile-first serving. Loss on SPA shells
    (some mobile sites return tiny bootstrap HTML).
  * am_prefix — `example.com` (no www) → `m.example.com`
  * drop_www — occasionally unblocks hosts that gate www but not apex.

Adding new transforms: prove they help on ≥2 unrelated sites first
(cross-site validation — bias check).
"""
from __future__ import annotations

from typing import Callable, Optional
from urllib.parse import urlsplit, urlunsplit


def _replace_host(url: str, new_host: str) -> str:
    parts = urlsplit(url)
    return urlunsplit(parts._replace(netloc=new_host))


def _original(url: str) -> Optional[str]:
    return url


def _mobile_subdomain(url: str) -> Optional[str]:
    """`https://www.example.com/a` → `https://m.example.com/a` (only if host starts with www.)."""
    parts = urlsplit(url)
    host = parts.hostname or ""
    if not host.startswith("www."):
        return None
    new_host = "m." + host[4:]
    if parts.port:
        new_host = f"{new_host}:{parts.port}"
    return _replace_host(url, new_host)


def _am_prefix(url: str) -> Optional[str]:
    """`https://example.com/a` → `https://m.example.com/a` (only if host has no subdomain)."""
    parts = urlsplit(url)
    host = parts.hostname or ""
    if not host or host.startswith("m."):
        return None
    # Only apply to apex-like hosts (≤2 dot-separated labels).
    if host.count(".") >= 2 and not host.startswith("www."):
        return None
    if host.startswith("www."):
        return None  # handled by mobile_subdomain
    return _replace_host(url, "m." + host)


def _drop_www(url: str) -> Optional[str]:
    parts = urlsplit(url)
    host = parts.hostname or ""
    if not host.startswith("www."):
        return None
    return _replace_host(url, host[4:])


TRANSFORMS: dict[str, Callable[[str], Optional[str]]] = {
    "original": _original,
    "mobile_subdomain": _mobile_subdomain,
    "am_prefix": _am_prefix,
    "drop_www": _drop_www,
}


def apply_transform(name: str, url: str) -> Optional[str]:
    """Apply one transform by name. Returns transformed URL or None if skipped."""
    fn = TRANSFORMS.get(name)
    if fn is None:
        raise ValueError(f"Unknown transform: {name!r}. Known: {list(TRANSFORMS)}")
    return fn(url)


def iter_transformed(url: str, order: list[str]) -> list[tuple[str, str]]:
    """Yield (transform_name, transformed_url) pairs for a given order.

    Skips transforms that return None (not applicable) and deduplicates
    URLs (so `original` and `drop_www` of `https://example.com` don't double-run).
    """
    seen: set[str] = set()
    out: list[tuple[str, str]] = []
    for name in order:
        new_url = apply_transform(name, url)
        if new_url is None:
            continue
        if new_url in seen:
            continue
        seen.add(new_url)
        out.append((name, new_url))
    return out
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall