Skip to content

JimLiu/baoyu-skills / skills/baoyu-url-to-markdown

baoyu-url-to-markdown

Verified

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

FileKindSize
SKILL.mdskill md6.1 KB
references/adapters.mdmarkdown2.5 KB
references/config/first-time-setup.mdmarkdown1.9 KB
references/quality-gate.mdmarkdown2.1 KB
scripts/baoyu-fetchexecutablescript122 B
scripts/bun.lockother34.6 KB
scripts/lib/adapters/generic/index.tsscript2.4 KB
scripts/lib/adapters/hn/index.tsscript10.2 KB
scripts/lib/adapters/index.tsscript834 B
scripts/lib/adapters/types.tsscript2.1 KB
scripts/lib/adapters/x/article.tsscript12.4 KB
scripts/lib/adapters/x/index.tsscript4.0 KB
scripts/lib/adapters/x/login.tsscript2.2 KB
scripts/lib/adapters/x/match.tsscript297 B
scripts/lib/adapters/x/payloads.tsscript1.6 KB
scripts/lib/adapters/x/session.tsscript1.4 KB
scripts/lib/adapters/x/shared.tsscript12.7 KB
scripts/lib/adapters/x/single.tsscript2.5 KB
scripts/lib/adapters/x/thread-loader.tsscript8.9 KB
scripts/lib/adapters/x/thread.tsscript9.0 KB
scripts/lib/adapters/x/types.tsscript600 B
scripts/lib/adapters/youtube/index.tsscript898 B
scripts/lib/adapters/youtube/transcript.tsscript13.0 KB
scripts/lib/adapters/youtube/utils.tsscript6.5 KB
scripts/lib/browser/cdp-client.tsscript6.7 KB
scripts/lib/browser/chrome-launcher.tsscript5.7 KB
scripts/lib/browser/cookie-sidecar.tsscript2.9 KB
scripts/lib/browser/interaction-gates.tsscript4.6 KB
scripts/lib/browser/network-journal.tsscript7.0 KB
scripts/lib/browser/page-snapshot.tsscript3.3 KB
scripts/lib/browser/profile.tsscript5.5 KB
scripts/lib/browser/session.tsscript4.5 KB
scripts/lib/cli.tsexecutablescript6.9 KB
scripts/lib/commands/convert.tsscript17.0 KB
scripts/lib/extract/document.tsscript846 B
scripts/lib/extract/html-cleaner.tsscript10.8 KB
scripts/lib/extract/html-extractor.tsscript2.2 KB
scripts/lib/extract/html-to-markdown.tsscript20.8 KB
scripts/lib/extract/markdown-renderer.tsscript4.6 KB
scripts/lib/media/default-downloader.tsscript5.1 KB
scripts/lib/media/markdown-media.tsscript13.3 KB
scripts/lib/media/media-utils.tsscript6.1 KB
scripts/lib/media/types.tsscript710 B
scripts/lib/types/defuddle-node.d.tsscript605 B
scripts/lib/types/shims.d.tsscript422 B
scripts/lib/utils/logger.tsscript645 B
scripts/lib/utils/url.tsscript300 B
scripts/package.jsonmanifest557 B

scripts/lib/adapters/youtube/utils.ts

View on GitHub
export interface YouTubeTranscriptSegment {
  start: number;
  end: number;
  text: string;
}

export interface YouTubeChapter {
  title: string;
  time: number;
}

interface RenderYouTubeTranscriptMarkdownInput {
  description?: string;
  segments: YouTubeTranscriptSegment[];
  chapters: YouTubeChapter[];
}

const DESCRIPTION_CHAPTER_RE = /^((?:\d{1,2}:)?\d{1,2}:\d{2})(?:\s+[-|:]\s+|\s+)(.+)$/;
const YOUTUBE_THUMBNAIL_VARIANTS = [
  "maxresdefault.jpg",
  "sddefault.jpg",
  "hqdefault.jpg",
  "mqdefault.jpg",
  "default.jpg",
];

export function isYouTubeHost(hostname: string): boolean {
  return [
    "youtube.com",
    "www.youtube.com",
    "m.youtube.com",
    "youtu.be",
  ].includes(hostname);
}

export function parseYouTubeVideoId(url: URL): string | null {
  if (url.hostname === "youtu.be") {
    return url.pathname.split("/").filter(Boolean)[0] ?? null;
  }

  if (url.pathname === "/watch") {
    return url.searchParams.get("v");
  }

  const shortsMatch = url.pathname.match(/^\/shorts\/([^/?#]+)/);
  if (shortsMatch) {
    return shortsMatch[1];
  }

  const liveMatch = url.pathname.match(/^\/live\/([^/?#]+)/);
  if (liveMatch) {
    return liveMatch[1];
  }

  const embedMatch = url.pathname.match(/^\/embed\/([^/?#]+)/);
  if (embedMatch) {
    return embedMatch[1];
  }

  return null;
}

function parseTimestampValue(raw: string): number | null {
  const parts = raw
    .split(":")
    .map((part) => Number.parseInt(part, 10))
    .filter((part) => Number.isFinite(part));

  if (parts.length < 2 || parts.length > 3) {
    return null;
  }

  if (parts.some((part) => part < 0)) {
    return null;
  }

  if (parts.length === 2) {
    const [minutes, seconds] = parts;
    return minutes * 60 + seconds;
  }

  const [hours, minutes, seconds] = parts;
  return hours * 3600 + minutes * 60 + seconds;
}

export function formatTimestamp(totalSeconds: number): string {
  const rounded = Math.max(0, Math.floor(totalSeconds));
  const hours = Math.floor(rounded / 3600);
  const minutes = Math.floor((rounded % 3600) / 60);
  const seconds = rounded % 60;

  if (hours > 0) {
    return `${hours}:${String(minutes).padStart(2, "0")}:${String(seconds).padStart(2, "0")}`;
  }
  return `${minutes}:${String(seconds).padStart(2, "0")}`;
}

export function formatTimestampRange(start: number, end: number): string {
  const safeStart = Math.max(0, start);
  const safeEnd = Math.max(safeStart, end);
  return `[${formatTimestamp(safeStart)} -> ${formatTimestamp(safeEnd)}]`;
}

export function normalizeYouTubeChapters(chapters: YouTubeChapter[]): YouTubeChapter[] {
  const seenTimes = new Set<number>();

  return chapters
    .map((chapter) => ({
      title: chapter.title.trim(),
      time: Math.max(0, Math.floor(chapter.time)),
    }))
    .filter((chapter) => chapter.title)
    .sort((left, right) => left.time - right.time)
    .filter((chapter) => {
      if (seenTimes.has(chapter.time)) {
        return false;
      }
      seenTimes.add(chapter.time);
      return true;
    });
}

export function parseYouTubeDescriptionChapters(description?: string | null): YouTubeChapter[] {
  if (!description) {
    return [];
  }

  const chapters: YouTubeChapter[] = [];
  const seen = new Set<string>();

  for (const rawLine of description.replace(/\r\n/g, "\n").split("\n")) {
    const line = rawLine.trim();
    if (!line) {
      continue;
    }

    const match = line.match(DESCRIPTION_CHAPTER_RE);
    if (!match) {
      continue;
    }

    const time = parseTimestampValue(match[1]);
    const title = match[2]?.trim();
    if (time === null || !title) {
      continue;
    }

    const key = `${time}:${title.toLowerCase()}`;
    if (seen.has(key)) {
      continue;
    }

    seen.add(key);
    chapters.push({ title, time });
  }

  const normalized = normalizeYouTubeChapters(chapters);
  if (normalized.length >= 2) {
    return normalized;
  }

  if (normalized.length === 1 && normalized[0]?.time === 0) {
    return normalized;
  }

  return [];
}

function renderDescriptionMarkdown(description: string): string {
  return description
    .replace(/\r\n/g, "\n")
    .trim()
    .split(/\n{2,}/)
    .map((block) => block.split("\n").map((line) => line.trimEnd()).join("  \n"))
    .join("\n\n")
    .trim();
}

function renderSegmentLine(segment: YouTubeTranscriptSegment): string {
  return `${formatTimestampRange(segment.start, segment.end)} ${segment.text}`;
}

export function renderYouTubeTranscriptMarkdown({
  description,
  segments,
  chapters,
}: RenderYouTubeTranscriptMarkdownInput): string {
  if (segments.length === 0) {
    return "";
  }

  const parts: string[] = [];
  const normalizedDescription = description?.trim();
  const transcriptEnd = segments.reduce((maxEnd, segment) => Math.max(maxEnd, segment.end, segment.start), 0);
  const normalizedChapters = normalizeYouTubeChapters(chapters).filter(
    (chapter) => transcriptEnd <= 0 || chapter.time < transcriptEnd,
  );

  if (normalizedDescription) {
    parts.push("## Description");
    parts.push(renderDescriptionMarkdown(normalizedDescription));
  }

  if (normalizedChapters.length > 0) {
    parts.push("## Chapters");

    for (let index = 0; index < normalizedChapters.length; index += 1) {
      const chapter = normalizedChapters[index];
      const nextChapter = normalizedChapters[index + 1];
      const chapterEnd = nextChapter ? nextChapter.time : transcriptEnd;
      const chapterSegments = segments.filter(
        (segment) => segment.start >= chapter.time && segment.start < chapterEnd,
      );

      parts.push(`### ${chapter.title} ${formatTimestampRange(chapter.time, chapterEnd)}`);

      if (chapterSegments.length > 0) {
        parts.push(chapterSegments.map(renderSegmentLine).join("\n"));
      }
    }
  } else {
    parts.push("## Transcript");
    parts.push(segments.map(renderSegmentLine).join("\n"));
  }

  return parts.filter(Boolean).join("\n\n").trim();
}

function normalizeThumbnailKey(url: string): string {
  try {
    const parsed = new URL(url);
    return `${parsed.origin}${parsed.pathname}`;
  } catch {
    return url;
  }
}

export function buildYouTubeThumbnailCandidates(videoId: string, listedUrls: string[]): string[] {
  const candidates = [
    ...YOUTUBE_THUMBNAIL_VARIANTS.map((variant) => `https://i.ytimg.com/vi/${videoId}/${variant}`),
    ...listedUrls,
  ];

  const seen = new Set<string>();
  return candidates.filter((candidate) => {
    if (!candidate) {
      return false;
    }

    const key = normalizeThumbnailKey(candidate);
    if (seen.has(key)) {
      return false;
    }

    seen.add(key);
    return true;
  });
}
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall