Skip to content

JimLiu/baoyu-skills / skills/baoyu-url-to-markdown

baoyu-url-to-markdown

Verified

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

FileKindSize
SKILL.mdskill md6.1 KB
references/adapters.mdmarkdown2.5 KB
references/config/first-time-setup.mdmarkdown1.9 KB
references/quality-gate.mdmarkdown2.1 KB
scripts/baoyu-fetchexecutablescript122 B
scripts/bun.lockother34.6 KB
scripts/lib/adapters/generic/index.tsscript2.4 KB
scripts/lib/adapters/hn/index.tsscript10.2 KB
scripts/lib/adapters/index.tsscript834 B
scripts/lib/adapters/types.tsscript2.1 KB
scripts/lib/adapters/x/article.tsscript12.4 KB
scripts/lib/adapters/x/index.tsscript4.0 KB
scripts/lib/adapters/x/login.tsscript2.2 KB
scripts/lib/adapters/x/match.tsscript297 B
scripts/lib/adapters/x/payloads.tsscript1.6 KB
scripts/lib/adapters/x/session.tsscript1.4 KB
scripts/lib/adapters/x/shared.tsscript12.7 KB
scripts/lib/adapters/x/single.tsscript2.5 KB
scripts/lib/adapters/x/thread-loader.tsscript8.9 KB
scripts/lib/adapters/x/thread.tsscript9.0 KB
scripts/lib/adapters/x/types.tsscript600 B
scripts/lib/adapters/youtube/index.tsscript898 B
scripts/lib/adapters/youtube/transcript.tsscript13.0 KB
scripts/lib/adapters/youtube/utils.tsscript6.5 KB
scripts/lib/browser/cdp-client.tsscript6.7 KB
scripts/lib/browser/chrome-launcher.tsscript5.7 KB
scripts/lib/browser/cookie-sidecar.tsscript2.9 KB
scripts/lib/browser/interaction-gates.tsscript4.6 KB
scripts/lib/browser/network-journal.tsscript7.0 KB
scripts/lib/browser/page-snapshot.tsscript3.3 KB
scripts/lib/browser/profile.tsscript5.5 KB
scripts/lib/browser/session.tsscript4.5 KB
scripts/lib/cli.tsexecutablescript6.9 KB
scripts/lib/commands/convert.tsscript17.0 KB
scripts/lib/extract/document.tsscript846 B
scripts/lib/extract/html-cleaner.tsscript10.8 KB
scripts/lib/extract/html-extractor.tsscript2.2 KB
scripts/lib/extract/html-to-markdown.tsscript20.8 KB
scripts/lib/extract/markdown-renderer.tsscript4.6 KB
scripts/lib/media/default-downloader.tsscript5.1 KB
scripts/lib/media/markdown-media.tsscript13.3 KB
scripts/lib/media/media-utils.tsscript6.1 KB
scripts/lib/media/types.tsscript710 B
scripts/lib/types/defuddle-node.d.tsscript605 B
scripts/lib/types/shims.d.tsscript422 B
scripts/lib/utils/logger.tsscript645 B
scripts/lib/utils/url.tsscript300 B
scripts/package.jsonmanifest557 B

scripts/lib/browser/page-snapshot.ts

View on GitHub
import type { BrowserSession } from "./session";

export interface CapturedPageSnapshot {
  html: string;
  finalUrl: string;
}

export const CAPTURE_NORMALIZED_PAGE_SCRIPT = String.raw`
(() => {
  const baseUrl = document.baseURI || location.href;
  const htmlClone = document.documentElement.cloneNode(true);

  function materializeShadowDom(sourceRoot, cloneRoot) {
    const sourceElements = Array.from(sourceRoot.querySelectorAll("*"));
    const cloneElements = Array.from(cloneRoot.querySelectorAll("*"));

    for (let index = sourceElements.length - 1; index >= 0; index -= 1) {
      const sourceElement = sourceElements[index];
      const cloneElement = cloneElements[index];
      const shadowRoot = sourceElement && sourceElement.shadowRoot;
      if (!shadowRoot || !cloneElement || !shadowRoot.innerHTML) {
        continue;
      }

      if (cloneElement.tagName && cloneElement.tagName.includes("-")) {
        const wrapper = document.createElement("div");
        wrapper.setAttribute("data-shadow-host", cloneElement.tagName.toLowerCase());
        wrapper.innerHTML = shadowRoot.innerHTML;
        cloneElement.replaceWith(wrapper);
      } else {
        cloneElement.innerHTML = shadowRoot.innerHTML;
      }
    }
  }

  function toAbsolute(url) {
    if (!url) return url;
    try {
      return new URL(url, baseUrl).href;
    } catch {
      return url;
    }
  }

  function absolutizeAttribute(root, selector, attribute) {
    root.querySelectorAll(selector).forEach((element) => {
      const value = element.getAttribute(attribute);
      if (!value) return;
      const absolute = toAbsolute(value);
      if (absolute) {
        element.setAttribute(attribute, absolute);
      }
    });
  }

  function absolutizeSrcset(root, selector) {
    root.querySelectorAll(selector).forEach((element) => {
      const srcset = element.getAttribute("srcset");
      if (!srcset) return;
      element.setAttribute(
        "srcset",
        srcset
          .split(",")
          .map((part) => {
            const trimmed = part.trim();
            if (!trimmed) return "";
            const [url, ...descriptor] = trimmed.split(/\s+/);
            const absolute = toAbsolute(url);
            return descriptor.length > 0 ? absolute + " " + descriptor.join(" ") : absolute;
          })
          .filter(Boolean)
          .join(", "),
      );
    });
  }

  materializeShadowDom(document.documentElement, htmlClone);

  htmlClone
    .querySelectorAll("img[data-src], video[data-src], audio[data-src], source[data-src]")
    .forEach((element) => {
      const dataSource = element.getAttribute("data-src");
      const current = element.getAttribute("src");
      if (dataSource && (!current || current === "" || current.startsWith("data:"))) {
        element.setAttribute("src", dataSource);
      }
    });

  absolutizeAttribute(htmlClone, "a[href]", "href");
  absolutizeAttribute(htmlClone, "img[src], video[src], audio[src], source[src], iframe[src]", "src");
  absolutizeAttribute(htmlClone, "video[poster]", "poster");
  absolutizeSrcset(htmlClone, "img[srcset], source[srcset]");

  return {
    html: "<!doctype html>\n" + htmlClone.outerHTML,
    finalUrl: location.href,
  };
})()
`;

export async function captureNormalizedPageSnapshot(
  browser: BrowserSession,
): Promise<CapturedPageSnapshot> {
  return browser.evaluate<CapturedPageSnapshot>(CAPTURE_NORMALIZED_PAGE_SCRIPT);
}
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall