Skip to content

JimLiu/baoyu-skills / skills/baoyu-url-to-markdown

baoyu-url-to-markdown

Verified

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

FileKindSize
SKILL.mdskill md6.1 KB
references/adapters.mdmarkdown2.5 KB
references/config/first-time-setup.mdmarkdown1.9 KB
references/quality-gate.mdmarkdown2.1 KB
scripts/baoyu-fetchexecutablescript122 B
scripts/bun.lockother34.6 KB
scripts/lib/adapters/generic/index.tsscript2.4 KB
scripts/lib/adapters/hn/index.tsscript10.2 KB
scripts/lib/adapters/index.tsscript834 B
scripts/lib/adapters/types.tsscript2.1 KB
scripts/lib/adapters/x/article.tsscript12.4 KB
scripts/lib/adapters/x/index.tsscript4.0 KB
scripts/lib/adapters/x/login.tsscript2.2 KB
scripts/lib/adapters/x/match.tsscript297 B
scripts/lib/adapters/x/payloads.tsscript1.6 KB
scripts/lib/adapters/x/session.tsscript1.4 KB
scripts/lib/adapters/x/shared.tsscript12.7 KB
scripts/lib/adapters/x/single.tsscript2.5 KB
scripts/lib/adapters/x/thread-loader.tsscript8.9 KB
scripts/lib/adapters/x/thread.tsscript9.0 KB
scripts/lib/adapters/x/types.tsscript600 B
scripts/lib/adapters/youtube/index.tsscript898 B
scripts/lib/adapters/youtube/transcript.tsscript13.0 KB
scripts/lib/adapters/youtube/utils.tsscript6.5 KB
scripts/lib/browser/cdp-client.tsscript6.7 KB
scripts/lib/browser/chrome-launcher.tsscript5.7 KB
scripts/lib/browser/cookie-sidecar.tsscript2.9 KB
scripts/lib/browser/interaction-gates.tsscript4.6 KB
scripts/lib/browser/network-journal.tsscript7.0 KB
scripts/lib/browser/page-snapshot.tsscript3.3 KB
scripts/lib/browser/profile.tsscript5.5 KB
scripts/lib/browser/session.tsscript4.5 KB
scripts/lib/cli.tsexecutablescript6.9 KB
scripts/lib/commands/convert.tsscript17.0 KB
scripts/lib/extract/document.tsscript846 B
scripts/lib/extract/html-cleaner.tsscript10.8 KB
scripts/lib/extract/html-extractor.tsscript2.2 KB
scripts/lib/extract/html-to-markdown.tsscript20.8 KB
scripts/lib/extract/markdown-renderer.tsscript4.6 KB
scripts/lib/media/default-downloader.tsscript5.1 KB
scripts/lib/media/markdown-media.tsscript13.3 KB
scripts/lib/media/media-utils.tsscript6.1 KB
scripts/lib/media/types.tsscript710 B
scripts/lib/types/defuddle-node.d.tsscript605 B
scripts/lib/types/shims.d.tsscript422 B
scripts/lib/utils/logger.tsscript645 B
scripts/lib/utils/url.tsscript300 B
scripts/package.jsonmanifest557 B

scripts/lib/cli.ts

View on GitHub
#!/usr/bin/env bun

import {
  runConvertCommand,
  type ConvertCommandOptions,
  type OutputFormat,
  type WaitMode,
} from "./commands/convert";

export const HELP_TEXT = `
baoyu-fetch - Read a URL into Markdown or JSON with Chrome CDP

Usage:
  baoyu-fetch <url> [options]

Options:
  --output <file>       Save output to file
  --format <type>       Output format: markdown | json
  --json                Alias for --format json
  --adapter <name>      Force an adapter (e.g. x, generic)
  --download-media      Download adapter-reported media into ./imgs and ./videos, then rewrite markdown links
  --media-dir <dir>     Base directory for downloaded media. Defaults to the output directory
  --debug-dir <dir>     Write debug artifacts
  --cdp-url <url>       Reuse an existing Chrome DevTools endpoint
  --browser-path <path> Explicit Chrome binary path
  --chrome-profile-dir <path>
                        Chrome user data dir. Defaults to BAOYU_CHROME_PROFILE_DIR
                        or baoyu-skills/chrome-profile.
  --headless            Launch a temporary headless Chrome if needed
  --wait-for <mode>     Wait mode: interaction | force
                        interaction: start visible Chrome and auto-wait only when login or verification is required
                        force: start visible Chrome, then auto-continue after it detects login/challenge progress
                               or continue immediately when you press Enter
  --wait-for-interaction
                        Alias for --wait-for interaction
  --wait-for-login      Alias for --wait-for interaction
  --interaction-timeout <ms>
                        How long to wait for manual interaction before failing (default: 600000)
  --interaction-poll-interval <ms>
                        How often to poll interaction state while waiting (default: 1500)
  --login-timeout <ms>  Alias for --interaction-timeout
  --login-poll-interval <ms>
                        Alias for --interaction-poll-interval
  --timeout <ms>        Page timeout in milliseconds (default: 30000)
  --help                Show help

Examples:
  baoyu-fetch https://example.com
  baoyu-fetch https://example.com --format markdown --output article.md --download-media
  baoyu-fetch https://example.com --format json --output article.json
  baoyu-fetch https://x.com/lennysan/status/2036483059407810640 --wait-for interaction
  baoyu-fetch https://x.com/lennysan/status/2036483059407810640 --wait-for force
`.trim();

interface CliOptions extends ConvertCommandOptions {
  url?: string;
  help: boolean;
}

function normalizeWaitMode(raw: string): WaitMode {
  const value = raw.toLowerCase();
  if (value === "interaction" || value === "auto") {
    return "interaction";
  }
  if (value === "force" || value === "manual" || value === "always") {
    return "force";
  }
  throw new Error(`Invalid wait mode: ${raw}. Expected interaction or force.`);
}

function normalizeOutputFormat(raw: string): OutputFormat {
  const value = raw.toLowerCase();
  if (value === "markdown" || value === "json") {
    return value;
  }

  throw new Error(`Invalid output format: ${raw}. Expected markdown or json.`);
}

export function parseArgs(argv: string[]): CliOptions {
  const options: CliOptions = {
    format: "markdown",
    headless: false,
    downloadMedia: false,
    waitMode: "none",
    interactionTimeoutMs: 600_000,
    interactionPollIntervalMs: 1_500,
    timeoutMs: 30_000,
    help: false,
  };

  const args = argv.slice(2);
  for (let index = 0; index < args.length; index += 1) {
    const value = args[index];

    if (value === "--help" || value === "-h") {
      options.help = true;
      continue;
    }
    if (value === "--format") {
      const format = args[index + 1];
      if (!format) {
        throw new Error("--format requires a value");
      }
      options.format = normalizeOutputFormat(format);
      index += 1;
      continue;
    }
    if (value === "--json") {
      options.format = "json";
      continue;
    }
    if (value === "--download-media") {
      options.downloadMedia = true;
      continue;
    }
    if (value === "--headless") {
      options.headless = true;
      continue;
    }
    if (value === "--wait-for") {
      const mode = args[index + 1];
      if (!mode) {
        throw new Error("--wait-for requires a mode");
      }
      options.waitMode = normalizeWaitMode(mode);
      index += 1;
      continue;
    }
    if (value === "--wait-for-interaction" || value === "--wait-for-login") {
      options.waitMode = "interaction";
      continue;
    }
    if (value === "--output") {
      options.output = args[index + 1];
      index += 1;
      continue;
    }
    if (value === "--adapter") {
      options.adapter = args[index + 1];
      index += 1;
      continue;
    }
    if (value === "--debug-dir") {
      options.debugDir = args[index + 1];
      index += 1;
      continue;
    }
    if (value === "--media-dir") {
      options.mediaDir = args[index + 1];
      index += 1;
      continue;
    }
    if (value === "--cdp-url") {
      options.cdpUrl = args[index + 1];
      index += 1;
      continue;
    }
    if (value === "--browser-path") {
      options.browserPath = args[index + 1];
      index += 1;
      continue;
    }
    if (value === "--chrome-profile-dir") {
      options.chromeProfileDir = args[index + 1];
      index += 1;
      continue;
    }
    if (value === "--timeout") {
      const parsed = Number(args[index + 1]);
      if (!Number.isFinite(parsed) || parsed <= 0) {
        throw new Error(`Invalid timeout: ${args[index + 1]}`);
      }
      options.timeoutMs = parsed;
      index += 1;
      continue;
    }
    if (value === "--interaction-timeout" || value === "--login-timeout") {
      const parsed = Number(args[index + 1]);
      if (!Number.isFinite(parsed) || parsed <= 0) {
        throw new Error(`Invalid interaction timeout: ${args[index + 1]}`);
      }
      options.interactionTimeoutMs = parsed;
      index += 1;
      continue;
    }
    if (value === "--interaction-poll-interval" || value === "--login-poll-interval") {
      const parsed = Number(args[index + 1]);
      if (!Number.isFinite(parsed) || parsed <= 0) {
        throw new Error(`Invalid interaction poll interval: ${args[index + 1]}`);
      }
      options.interactionPollIntervalMs = parsed;
      index += 1;
      continue;
    }
    if (value.startsWith("-")) {
      throw new Error(`Unknown option: ${value}`);
    }
    if (!options.url) {
      options.url = value;
      continue;
    }
    throw new Error(`Unexpected argument: ${value}`);
  }

  return options;
}

async function main(): Promise<void> {
  try {
    const options = parseArgs(process.argv);
    if (options.help || !options.url) {
      console.log(HELP_TEXT);
      return;
    }

    await runConvertCommand(options);
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    console.error(message);
    process.exitCode = 1;
  }
}

if (import.meta.main) {
  void main();
}
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall