Skip to content

JimLiu/baoyu-skills / skills/baoyu-url-to-markdown

baoyu-url-to-markdown

Verified

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

FileKindSize
SKILL.mdskill md6.1 KB
references/adapters.mdmarkdown2.5 KB
references/config/first-time-setup.mdmarkdown1.9 KB
references/quality-gate.mdmarkdown2.1 KB
scripts/baoyu-fetchexecutablescript122 B
scripts/bun.lockother34.6 KB
scripts/lib/adapters/generic/index.tsscript2.4 KB
scripts/lib/adapters/hn/index.tsscript10.2 KB
scripts/lib/adapters/index.tsscript834 B
scripts/lib/adapters/types.tsscript2.1 KB
scripts/lib/adapters/x/article.tsscript12.4 KB
scripts/lib/adapters/x/index.tsscript4.0 KB
scripts/lib/adapters/x/login.tsscript2.2 KB
scripts/lib/adapters/x/match.tsscript297 B
scripts/lib/adapters/x/payloads.tsscript1.6 KB
scripts/lib/adapters/x/session.tsscript1.4 KB
scripts/lib/adapters/x/shared.tsscript12.7 KB
scripts/lib/adapters/x/single.tsscript2.5 KB
scripts/lib/adapters/x/thread-loader.tsscript8.9 KB
scripts/lib/adapters/x/thread.tsscript9.0 KB
scripts/lib/adapters/x/types.tsscript600 B
scripts/lib/adapters/youtube/index.tsscript898 B
scripts/lib/adapters/youtube/transcript.tsscript13.0 KB
scripts/lib/adapters/youtube/utils.tsscript6.5 KB
scripts/lib/browser/cdp-client.tsscript6.7 KB
scripts/lib/browser/chrome-launcher.tsscript5.7 KB
scripts/lib/browser/cookie-sidecar.tsscript2.9 KB
scripts/lib/browser/interaction-gates.tsscript4.6 KB
scripts/lib/browser/network-journal.tsscript7.0 KB
scripts/lib/browser/page-snapshot.tsscript3.3 KB
scripts/lib/browser/profile.tsscript5.5 KB
scripts/lib/browser/session.tsscript4.5 KB
scripts/lib/cli.tsexecutablescript6.9 KB
scripts/lib/commands/convert.tsscript17.0 KB
scripts/lib/extract/document.tsscript846 B
scripts/lib/extract/html-cleaner.tsscript10.8 KB
scripts/lib/extract/html-extractor.tsscript2.2 KB
scripts/lib/extract/html-to-markdown.tsscript20.8 KB
scripts/lib/extract/markdown-renderer.tsscript4.6 KB
scripts/lib/media/default-downloader.tsscript5.1 KB
scripts/lib/media/markdown-media.tsscript13.3 KB
scripts/lib/media/media-utils.tsscript6.1 KB
scripts/lib/media/types.tsscript710 B
scripts/lib/types/defuddle-node.d.tsscript605 B
scripts/lib/types/shims.d.tsscript422 B
scripts/lib/utils/logger.tsscript645 B
scripts/lib/utils/url.tsscript300 B
scripts/package.jsonmanifest557 B

scripts/lib/browser/interaction-gates.ts

View on GitHub
import type { WaitForInteractionRequest } from "../adapters/types";
import type { BrowserSession } from "./session";

interface GateSnapshot {
  title: string;
  currentUrl: string;
  bodyText: string;
  hasCloudflareTurnstile: boolean;
  hasCloudflareChallenge: boolean;
  hasRecaptcha: boolean;
  hasRecaptchaIframe: boolean;
  hasHcaptcha: boolean;
  hasHcaptchaIframe: boolean;
}

export function detectInteractionGateFromSnapshot(snapshot: GateSnapshot): WaitForInteractionRequest | null {
  const text = snapshot.bodyText.toLowerCase();
  const title = snapshot.title.toLowerCase();
  const url = snapshot.currentUrl.toLowerCase();

  if (
    snapshot.hasCloudflareTurnstile ||
    snapshot.hasCloudflareChallenge ||
    title.includes("just a moment") ||
    text.includes("verify you are human") ||
    text.includes("checking your browser before accessing") ||
    text.includes("enable javascript and cookies to continue") ||
    url.includes("/cdn-cgi/challenge-platform/")
  ) {
    return {
      type: "wait_for_interaction",
      kind: "cloudflare",
      provider: "cloudflare",
      reason: "Cloudflare human verification detected",
      prompt: "Please complete the Cloudflare verification in the opened Chrome window. Extraction will continue automatically once the challenge disappears.",
      requiresVisibleBrowser: true,
    };
  }

  if (
    snapshot.hasRecaptcha ||
    snapshot.hasRecaptchaIframe ||
    text.includes("i'm not a robot")
  ) {
    return {
      type: "wait_for_interaction",
      kind: "recaptcha",
      provider: "google_recaptcha",
      reason: "Google reCAPTCHA detected",
      prompt: "Please complete the reCAPTCHA verification in the opened Chrome window. Extraction will continue automatically once the challenge disappears.",
      requiresVisibleBrowser: true,
    };
  }

  if (
    snapshot.hasHcaptcha ||
    snapshot.hasHcaptchaIframe ||
    text.includes("hcaptcha")
  ) {
    return {
      type: "wait_for_interaction",
      kind: "hcaptcha",
      provider: "hcaptcha",
      reason: "hCaptcha verification detected",
      prompt: "Please complete the hCaptcha verification in the opened Chrome window. Extraction will continue automatically once the challenge disappears.",
      requiresVisibleBrowser: true,
    };
  }

  return null;
}

export async function detectInteractionGate(browser: BrowserSession): Promise<WaitForInteractionRequest | null> {
  const snapshot = await browser.evaluate<GateSnapshot>(`
    (() => {
      const bodyText = (document.body?.innerText ?? "").slice(0, 4000);
      return {
        title: document.title ?? "",
        currentUrl: window.location.href,
        bodyText,
        hasCloudflareTurnstile: Boolean(
          document.querySelector(
            '.cf-turnstile, [name="cf-turnstile-response"], iframe[src*="challenges.cloudflare.com"]'
          )
        ),
        hasCloudflareChallenge: Boolean(
          document.querySelector(
            '#challenge-running, #cf-challenge-running, .challenge-platform, [data-ray], [data-translate="checking_browser"]'
          )
        ),
        hasRecaptcha: Array.from(
          document.querySelectorAll('.g-recaptcha, iframe[title*="reCAPTCHA"]')
        ).some((el) => {
          if (el.closest(".grecaptcha-badge")) return false;
          if (el.getAttribute("data-size") === "invisible") return false;
          const src = el.getAttribute("src") ?? "";
          if (src.includes("size=invisible")) return false;
          if (src.includes("/bframe")) return false;
          return true;
        }),
        hasRecaptchaIframe: Array.from(
          document.querySelectorAll('iframe[src*="recaptcha/api2/bframe"], iframe[src*="recaptcha/enterprise/bframe"]')
        ).some((el) => {
          const style = window.getComputedStyle(el);
          if (style.display === "none" || style.visibility === "hidden" || Number(style.opacity) === 0) return false;
          const rect = el.getBoundingClientRect();
          return rect.width > 0 && rect.height > 0 && rect.bottom > 0 && rect.right > 0;
        }),
        hasHcaptcha: Boolean(
          document.querySelector(
            '.h-captcha, textarea[name="h-captcha-response"], iframe[title*="hCaptcha"]'
          )
        ),
        hasHcaptchaIframe: Boolean(
          document.querySelector('iframe[src*="hcaptcha.com"]')
        ),
      };
    })()
  `).catch(() => ({
    title: "",
    currentUrl: "",
    bodyText: "",
    hasCloudflareTurnstile: false,
    hasCloudflareChallenge: false,
    hasRecaptcha: false,
    hasRecaptchaIframe: false,
    hasHcaptcha: false,
    hasHcaptchaIframe: false,
  }));

  return detectInteractionGateFromSnapshot(snapshot);
}
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall