Skip to content

code-yeongyu/oh-my-openagent / packages/shared-skills/skills/ultimate-browsing

ultimate-browsing

Flagged

Reaches web pages the user's browser cannot open: WAF-blocked and bot-scored hosts, platform-native readers (Chinese and social platforms), archive snapshots with provenance, and a browser you own for stealth or throwaway profiles. Not for plain search, an unblocked static fetch, or a page the user's browser can open.

FileKindSize
.gitignoreother29 B
ATTRIBUTION.mdmarkdown2.5 KB
SKILL.mdskill md11.9 KB
engine/AGENTS.mdmarkdown10.8 KB
engine/__init__.pyscript668 B
engine/__main__.pyscript5.0 KB
engine/bias_check.pyscript7.1 KB
engine/curl_probe.pyscript2.4 KB
engine/executor.pyscript8.0 KB
engine/fetch_chain.pyscript13.0 KB
engine/referers.pyscript327 B
engine/result_schema.pyscript1.5 KB
engine/summary.pyscript1.2 KB
engine/surrogate.pyscript7.9 KB
engine/surrogates.yamlconfig2.3 KB
engine/templates/knip.workspace.jsoncother1.5 KB
engine/templates/package.jsonmanifest349 B
engine/templates/playwright_mobile_chrome.jsscript3.6 KB
engine/templates/playwright_real_chrome.jsscript5.5 KB
engine/tests/fixtures/amp_redirect_stub.htmlother323 B
engine/tests/fixtures/search_interstitial.htmlother90.4 KB
engine/tests/fixtures/wayback_available.jsonconfig246 B
engine/tests/fixtures/wayback_snapshot.htmlother181.4 KB
engine/tests/test_executor_timeout.pyscript6.2 KB
engine/tests/test_fetch_chain.pyscript2.7 KB
engine/tests/test_playwright_stealth.pyscript3.9 KB
engine/tests/test_playwright_templates.pyscript10.7 KB
engine/tests/test_surrogate.pyscript11.3 KB
engine/tests/test_surrogate_validators.pyscript3.5 KB
engine/url_transforms.pyscript3.2 KB
engine/validators.pyscript9.1 KB
engine/waf_detector.pyscript6.9 KB
engine/waf_profiles.yamlconfig6.2 KB
references/agent-reach/README.mdmarkdown2.2 KB
references/agent-reach/career.mdmarkdown755 B
references/agent-reach/dev.mdmarkdown1.3 KB
references/agent-reach/search.mdmarkdown821 B
references/agent-reach/social.mdmarkdown5.6 KB
references/agent-reach/video.mdmarkdown2.7 KB
references/agent-reach/web.mdmarkdown2.1 KB
references/chrome-stealth.mdmarkdown2.4 KB
references/insane-search/README.mdmarkdown20.2 KB
references/insane-search/cache-archive.mdmarkdown4.1 KB
references/insane-search/fallback.mdmarkdown6.9 KB
references/insane-search/jina.mdmarkdown4.3 KB
references/insane-search/json-api.mdmarkdown3.5 KB
references/insane-search/media.mdmarkdown3.5 KB
references/insane-search/metadata.mdmarkdown2.9 KB
references/insane-search/naver.mdmarkdown3.4 KB
references/insane-search/playwright.mdmarkdown6.8 KB
references/insane-search/public-api.mdmarkdown3.4 KB
references/insane-search/rss.mdmarkdown3.0 KB
references/insane-search/tls-impersonate.mdmarkdown6.5 KB
references/insane-search/twitter.mdmarkdown3.5 KB
scripts/cookie_crypto.pyscript2.9 KB
scripts/cookie_domains.pyscript1.1 KB
scripts/cookie_paths.pyscript3.8 KB
scripts/extract_cookies.pyscript9.4 KB
scripts/tests/test_cookie_domain_filter.pyscript3.5 KB
scripts/tests/test_extract_cookies.pyscript9.4 KB

references/insane-search/jina.md

View on GitHub
# 범용 웹 추출 — Jina Reader

> `r.jina.ai/URL` 한 줄로 거의 모든 공개 URL을 마크다운으로 변환.
> Puppeteer 기반 실제 브라우저 렌더링 — JS SPA까지 처리.
>
> **2026-08-09 probe 기준 무료 무키 경로는 종료됨.** 익명 호출은 401이며,
> 리다이렉트를 따라가면 Cloudflare Turnstile(`Just a moment...`)에 막힌다.
> **이제 `JINA_API_KEY` 환경 변수가 필요**하다 — `Authorization: Bearer <key>` 헤더.
> 엔진에서는 `engine/surrogates.yaml`의 `jina_reader` 엔트리가 키가 있을 때만 활성화된다
> (kind=reader, provenance=live — 서버 측 재렌더링).
> 예전 "무료 500 RPM" 안내는 모두 폐기되었으므로 따르지 않는다.

## 기본 사용

```bash
curl -s -H "Authorization: Bearer ${JINA_API_KEY}" "https://r.jina.ai/{URL}"
```

## 고급 기능

### JSON 구조화 출력

```bash
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "Accept: application/json" "https://r.jina.ai/{URL}"
```

반환: `data.{title, description, url, content, metadata, external, usage}`

**핵심**: `external.alternate`에서 사이트의 **RSS URL을 자동 발견** 가능.

### CSS 선택자 타겟팅

```bash
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-Target-Selector: .article-body" "https://r.jina.ai/{URL}"
```

네비게이션/풋터 제거, 본문만 추출. 커뮤니티 게시판에서 특히 효과적.

### SPA 스트리밍 모드

```bash
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "Accept: text/event-stream" "https://r.jina.ai/{URL}"
```

JS 로딩 완료까지 대기. 동적 콘텐츠가 완전히 렌더링된 최종 버전 반환.

### 스크린샷

```bash
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-Respond-With: screenshot" "https://r.jina.ai/{URL}"
```

GCS 서명 URL 반환 (4시간 유효). 비주얼 검증 용도.

### PDF 처리

```bash
curl -s -H "Authorization: Bearer ${JINA_API_KEY}" "https://r.jina.ai/https://example.com/file.pdf"
```

PDF → 마크다운 자동 변환. 페이지 수 메타데이터 포함.

### 쿠키 전달 (인증 사이트)

```bash
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-Set-Cookie: session=abc123" "https://r.jina.ai/{URL}"
```

### 링크 보존

```bash
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-With-Links: true" "https://r.jina.ai/{URL}"
```

### 캐시 제어

```bash
# 캐시 우회 (실시간 필요 시)
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-No-Cache: true" "https://r.jina.ai/{URL}"

# 캐시 TTL 지정 (초)
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-Cache-Tolerance: 600" "https://r.jina.ai/{URL}"
```

### 순수 텍스트 / 원본 HTML

```bash
# body.innerText만
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-Respond-With: text" "https://r.jina.ai/{URL}"

# 원본 HTML
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "X-Respond-With: html" "https://r.jina.ai/{URL}"
```

## 검증된 성공 사이트

| 사이트 | 결과 | 비고 |
|--------|------|------|
| Threads | 성공 | 프로필 + 포스트 |
| 클리앙 | 성공 | 게시글 목록 + 본문 |
| 루리웹 | 성공 | 게시글 목록 + 본문 |
| 뽐뿌 | 성공 | 게시글 + RSS도 가능 |
| 네이버 뉴스 | 성공 | 기사 목록 + 본문 완전 |
| 네이버 증권 | 성공 | 실시간 주가 |
| 긱뉴스 | 성공 | 토픽 목록 + 본문 |
| 44bits | 성공 | 기사 목록 |
| 커리어리 | 성공 | JS 렌더링으로 추출 |
| 브런치 | 성공 | 기사 전문 |
| 한경 | 성공 | 뉴스 기사 |
| 다음 뉴스 | 성공 | 뉴스 기사 |
| Medium | 성공 | 기사 전문 (paywall 제외) |
| Substack | 성공 | 뉴스레터 전문 |
| dev.to | 성공 | 기사 전문 |
| PDF (모든 URL) | 성공 | 자동 변환 |

## 실패하는 사이트

| 사이트 | 이유 |
|--------|------|
| X/Twitter | 402 — Syndication/oEmbed 사용 (twitter.md 참조) |
| Reddit | 차단 — JSON API 사용 (json-api.md 참조) |
| 디시인사이드 | 빈 본문 반환 |
| 에펨코리아 | HTTP 430 |
| 요즘IT | CloudFront 403 |
| 네이버 쇼핑 | CAPTCHA |
| 쿠팡 | WAF 차단 |


## RSS 자동 발견

Jina JSON 모드의 `external.alternate`에서 사이트의 RSS URL이 자동 노출됨:

```bash
curl -H "Authorization: Bearer ${JINA_API_KEY}" -H "Accept: application/json" "https://r.jina.ai/{URL}" | \
python3 -c "import sys,json; print(json.load(sys.stdin)['data'].get('external',{}))"
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall