Skip to content

JimLiu/baoyu-skills / skills/baoyu-url-to-markdown

baoyu-url-to-markdown

Verified

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

FileKindSize
SKILL.mdskill md6.1 KB
references/adapters.mdmarkdown2.5 KB
references/config/first-time-setup.mdmarkdown1.9 KB
references/quality-gate.mdmarkdown2.1 KB
scripts/baoyu-fetchexecutablescript122 B
scripts/bun.lockother34.6 KB
scripts/lib/adapters/generic/index.tsscript2.4 KB
scripts/lib/adapters/hn/index.tsscript10.2 KB
scripts/lib/adapters/index.tsscript834 B
scripts/lib/adapters/types.tsscript2.1 KB
scripts/lib/adapters/x/article.tsscript12.4 KB
scripts/lib/adapters/x/index.tsscript4.0 KB
scripts/lib/adapters/x/login.tsscript2.2 KB
scripts/lib/adapters/x/match.tsscript297 B
scripts/lib/adapters/x/payloads.tsscript1.6 KB
scripts/lib/adapters/x/session.tsscript1.4 KB
scripts/lib/adapters/x/shared.tsscript12.7 KB
scripts/lib/adapters/x/single.tsscript2.5 KB
scripts/lib/adapters/x/thread-loader.tsscript8.9 KB
scripts/lib/adapters/x/thread.tsscript9.0 KB
scripts/lib/adapters/x/types.tsscript600 B
scripts/lib/adapters/youtube/index.tsscript898 B
scripts/lib/adapters/youtube/transcript.tsscript13.0 KB
scripts/lib/adapters/youtube/utils.tsscript6.5 KB
scripts/lib/browser/cdp-client.tsscript6.7 KB
scripts/lib/browser/chrome-launcher.tsscript5.7 KB
scripts/lib/browser/cookie-sidecar.tsscript2.9 KB
scripts/lib/browser/interaction-gates.tsscript4.6 KB
scripts/lib/browser/network-journal.tsscript7.0 KB
scripts/lib/browser/page-snapshot.tsscript3.3 KB
scripts/lib/browser/profile.tsscript5.5 KB
scripts/lib/browser/session.tsscript4.5 KB
scripts/lib/cli.tsexecutablescript6.9 KB
scripts/lib/commands/convert.tsscript17.0 KB
scripts/lib/extract/document.tsscript846 B
scripts/lib/extract/html-cleaner.tsscript10.8 KB
scripts/lib/extract/html-extractor.tsscript2.2 KB
scripts/lib/extract/html-to-markdown.tsscript20.8 KB
scripts/lib/extract/markdown-renderer.tsscript4.6 KB
scripts/lib/media/default-downloader.tsscript5.1 KB
scripts/lib/media/markdown-media.tsscript13.3 KB
scripts/lib/media/media-utils.tsscript6.1 KB
scripts/lib/media/types.tsscript710 B
scripts/lib/types/defuddle-node.d.tsscript605 B
scripts/lib/types/shims.d.tsscript422 B
scripts/lib/utils/logger.tsscript645 B
scripts/lib/utils/url.tsscript300 B
scripts/package.jsonmanifest557 B

scripts/lib/browser/session.ts

View on GitHub
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { CdpClient, TargetSession, evaluateRuntime } from "./cdp-client";

interface NavigationResult {
  errorText?: string;
}

const execFileAsync = promisify(execFile);
const MACOS_BROWSER_APP_IDS = [
  "com.google.Chrome",
  "org.chromium.Chromium",
  "com.brave.Browser",
  "com.microsoft.edgemac",
];

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

async function activateBrowserApp(): Promise<void> {
  if (process.platform !== "darwin") {
    return;
  }

  for (const appId of MACOS_BROWSER_APP_IDS) {
    try {
      await execFileAsync("osascript", ["-e", `tell application id "${appId}" to activate`]);
      return;
    } catch {
      // Try the next installed browser bundle id.
    }
  }
}

export class BrowserSession {
  private constructor(
    private readonly cdp: CdpClient,
    public readonly targetSession: TargetSession,
    public readonly interactive: boolean,
  ) {}

  static async open(
    cdp: CdpClient,
    options: {
      initialUrl?: string;
      interactive?: boolean;
    } = {},
  ): Promise<BrowserSession> {
    const targetSession = await cdp.createPageSession({
      initialUrl: options.initialUrl,
      visible: options.interactive,
    });
    const browser = new BrowserSession(cdp, targetSession, Boolean(options.interactive));
    if (browser.interactive) {
      await browser.bringToFront().catch(() => {});
    }
    return browser;
  }

  async goto(url: string, timeoutMs = 30_000): Promise<void> {
    const loadPromise = this.targetSession.waitForEvent("Page.loadEventFired", undefined, timeoutMs).catch(() => null);
    const result = await this.targetSession.send<NavigationResult>("Page.navigate", { url });
    if (result.errorText) {
      throw new Error(`Navigation failed: ${result.errorText}`);
    }
    await loadPromise;
    await this.waitForReadyState(timeoutMs);
  }

  async waitForReadyState(timeoutMs = 30_000): Promise<void> {
    const startedAt = Date.now();
    while (Date.now() - startedAt < timeoutMs) {
      const state = await this.evaluate<string>("document.readyState");
      if (state === "interactive" || state === "complete") {
        return;
      }
      await sleep(150);
    }
    throw new Error("Timed out waiting for document.readyState");
  }

  async evaluate<T>(expression: string): Promise<T> {
    return evaluateRuntime<T>(this.targetSession, expression);
  }

  async getHTML(): Promise<string> {
    return this.evaluate<string>("document.documentElement.outerHTML");
  }

  async getTitle(): Promise<string> {
    return this.evaluate<string>("document.title");
  }

  async getURL(): Promise<string> {
    return this.evaluate<string>("window.location.href");
  }

  async bringToFront(): Promise<void> {
    await this.targetSession.send("Page.bringToFront").catch(async () => {
      await this.cdp.sendBrowserCommand("Target.activateTarget", {
        targetId: this.targetSession.targetId,
      });
    });
    if (this.interactive) {
      await activateBrowserApp().catch(() => {});
    }
  }

  async click(selector: string): Promise<void> {
    const result = await this.evaluate<{ ok: boolean; error?: string }>(`
      (() => {
        const element = document.querySelector(${JSON.stringify(selector)});
        if (!element) {
          return { ok: false, error: "Element not found" };
        }
        element.scrollIntoView({ block: "center", inline: "center" });
        if (element instanceof HTMLElement) {
          element.click();
          return { ok: true };
        }
        return { ok: false, error: "Element is not clickable" };
      })()
    `);

    if (!result.ok) {
      throw new Error(result.error ?? `Failed to click ${selector}`);
    }
  }

  async scrollToEnd(options: { stepPx?: number; delayMs?: number; maxSteps?: number } = {}): Promise<void> {
    const stepPx = options.stepPx ?? 1_400;
    const delayMs = options.delayMs ?? 250;
    const maxSteps = options.maxSteps ?? 6;

    for (let step = 0; step < maxSteps; step += 1) {
      const done = await this.evaluate<boolean>(`
        (() => {
          const before = window.scrollY;
          window.scrollBy(0, ${stepPx});
          const atBottom = window.innerHeight + window.scrollY >= document.body.scrollHeight - 4;
          return atBottom || window.scrollY === before;
        })()
      `);
      if (done) {
        break;
      }
      await sleep(delayMs);
    }
  }

  async close(): Promise<void> {
    await this.cdp.closeTarget(this.targetSession.targetId);
  }
}
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall