Skip to content

code-yeongyu/oh-my-openagent / packages/shared-skills/skills/ultimate-browsing

ultimate-browsing

Flagged

Reaches web pages the user's browser cannot open: WAF-blocked and bot-scored hosts, platform-native readers (Chinese and social platforms), archive snapshots with provenance, and a browser you own for stealth or throwaway profiles. Not for plain search, an unblocked static fetch, or a page the user's browser can open.

FileKindSize
.gitignoreother29 B
ATTRIBUTION.mdmarkdown2.5 KB
SKILL.mdskill md11.9 KB
engine/AGENTS.mdmarkdown10.8 KB
engine/__init__.pyscript668 B
engine/__main__.pyscript5.0 KB
engine/bias_check.pyscript7.1 KB
engine/curl_probe.pyscript2.4 KB
engine/executor.pyscript8.0 KB
engine/fetch_chain.pyscript13.0 KB
engine/referers.pyscript327 B
engine/result_schema.pyscript1.5 KB
engine/summary.pyscript1.2 KB
engine/surrogate.pyscript7.9 KB
engine/surrogates.yamlconfig2.3 KB
engine/templates/knip.workspace.jsoncother1.5 KB
engine/templates/package.jsonmanifest349 B
engine/templates/playwright_mobile_chrome.jsscript3.6 KB
engine/templates/playwright_real_chrome.jsscript5.5 KB
engine/tests/fixtures/amp_redirect_stub.htmlother323 B
engine/tests/fixtures/search_interstitial.htmlother90.4 KB
engine/tests/fixtures/wayback_available.jsonconfig246 B
engine/tests/fixtures/wayback_snapshot.htmlother181.4 KB
engine/tests/test_executor_timeout.pyscript6.2 KB
engine/tests/test_fetch_chain.pyscript2.7 KB
engine/tests/test_playwright_stealth.pyscript3.9 KB
engine/tests/test_playwright_templates.pyscript10.7 KB
engine/tests/test_surrogate.pyscript11.3 KB
engine/tests/test_surrogate_validators.pyscript3.5 KB
engine/url_transforms.pyscript3.2 KB
engine/validators.pyscript9.1 KB
engine/waf_detector.pyscript6.9 KB
engine/waf_profiles.yamlconfig6.2 KB
references/agent-reach/README.mdmarkdown2.2 KB
references/agent-reach/career.mdmarkdown755 B
references/agent-reach/dev.mdmarkdown1.3 KB
references/agent-reach/search.mdmarkdown821 B
references/agent-reach/social.mdmarkdown5.6 KB
references/agent-reach/video.mdmarkdown2.7 KB
references/agent-reach/web.mdmarkdown2.1 KB
references/chrome-stealth.mdmarkdown2.4 KB
references/insane-search/README.mdmarkdown20.2 KB
references/insane-search/cache-archive.mdmarkdown4.1 KB
references/insane-search/fallback.mdmarkdown6.9 KB
references/insane-search/jina.mdmarkdown4.3 KB
references/insane-search/json-api.mdmarkdown3.5 KB
references/insane-search/media.mdmarkdown3.5 KB
references/insane-search/metadata.mdmarkdown2.9 KB
references/insane-search/naver.mdmarkdown3.4 KB
references/insane-search/playwright.mdmarkdown6.8 KB
references/insane-search/public-api.mdmarkdown3.4 KB
references/insane-search/rss.mdmarkdown3.0 KB
references/insane-search/tls-impersonate.mdmarkdown6.5 KB
references/insane-search/twitter.mdmarkdown3.5 KB
scripts/cookie_crypto.pyscript2.9 KB
scripts/cookie_domains.pyscript1.1 KB
scripts/cookie_paths.pyscript3.8 KB
scripts/extract_cookies.pyscript9.4 KB
scripts/tests/test_cookie_domain_filter.pyscript3.5 KB
scripts/tests/test_extract_cookies.pyscript9.4 KB

references/insane-search/metadata.md

View on GitHub
# 메타데이터 추출 — OGP / JSON-LD / Schema.org

> HTML을 받았을 때 구조화된 데이터를 추출하는 보조 기법.
> 본문 전체를 못 가져와도 제목, 요약, 가격, 프로필 등 핵심 정보를 확보할 수 있다.

## 의존성

없음 (curl + python3 기본 모듈).

## OGP (Open Graph Protocol) 메타태그

대부분의 사이트가 소셜 공유용으로 삽입. 제목 + 설명 + 이미지 확보 가능.

```bash
curl -sL -H "User-Agent: Mozilla/5.0 ..." "{URL}" | \
  python3 -c "
import sys, re
html = sys.stdin.read()
for m in re.findall(r'<meta property=\"og:(\w+)\" content=\"([^\"]*?)\"', html):
    print(f'og:{m[0]} = {m[1]}')
for m in re.findall(r'<meta name=\"description\" content=\"([^\"]*?)\"', html):
    print(f'description = {m}')
"
```

## JSON-LD (Schema.org 구조화 데이터)

**가장 가치 높은 추출 대상.** 상품, 기사, 프로필 등 구조화된 정보가 JSON으로 들어있다.

```bash
curl -sL "{URL}" | \
  python3 -c "
import sys, re, json
html = sys.stdin.read()
blocks = re.findall(r'<script type=\"application/ld\+json\">(.*?)</script>', html, re.DOTALL)
for b in blocks:
    try:
        data = json.loads(b)
        print(json.dumps(data, ensure_ascii=False, indent=2))
    except:
        pass
"
```

### 실제 사례

**쿠팡 검색 결과** — `CollectionPage` + `ItemList`:
```json
{
  "@type": "CollectionPage",
  "mainEntity": {
    "@type": "ItemList",
    "itemListElement": [
      {
        "@type": "ListItem",
        "item": {
          "@type": "Product",
          "name": "...",
          "offers": { "price": 29900 }
        }
      }
    ]
  }
}
```

**LinkedIn 프로필** — `Person`:
```json
{
  "@type": "Person",
  "name": "...",
  "jobTitle": "...",
  "alumniOf": [
    { "@type": "Organization", "name": "..." }
  ]
}
```

**뉴스 기사** — `NewsArticle`:
```json
{
  "@type": "NewsArticle",
  "headline": "...",
  "datePublished": "2026-04-16",
  "author": { "name": "..." },
  "articleBody": "..."
}
```

## Next.js RSC 페이로드 (요즘IT 등)

Next.js App Router 사이트는 `self.__next_f.push()` 스크립트에 콘텐츠가 포함됨.

```bash
curl -sL "{URL}" | \
  python3 -c "
import sys, re
html = sys.stdin.read()
chunks = re.findall(r'self\.__next_f\.push\(\[1,\"(.*?)\"\]\)', html)
text = ''.join(chunks)
# 한국어 텍스트 추출 (유니코드 이스케이프 디코딩)
decoded = text.encode().decode('unicode_escape', errors='ignore')
print(decoded[:3000])
"
```

## 활용 시점

메타데이터 추출은 **독립 방법이 아니라 보조 기법**이다.
어떤 Phase에서든 HTML을 받으면 같이 실행:

- Phase 1에서 curl로 HTML 받음 → JSON-LD도 추출
- Phase 2에서 curl_cffi로 HTML 받음 → JSON-LD도 추출
- Phase 3에서 Playwright로 DOM 받음 → `browser_evaluate`로 JSON-LD 추출

본문은 못 가져와도 JSON-LD에서 **상품 가격, 기사 요약, 프로필 정보**는 확보될 수 있다.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall