Skip to content

JimLiu/baoyu-skills / skills/baoyu-url-to-markdown

baoyu-url-to-markdown

Verified

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

FileKindSize
SKILL.mdskill md6.1 KB
references/adapters.mdmarkdown2.5 KB
references/config/first-time-setup.mdmarkdown1.9 KB
references/quality-gate.mdmarkdown2.1 KB
scripts/baoyu-fetchexecutablescript122 B
scripts/bun.lockother34.6 KB
scripts/lib/adapters/generic/index.tsscript2.4 KB
scripts/lib/adapters/hn/index.tsscript10.2 KB
scripts/lib/adapters/index.tsscript834 B
scripts/lib/adapters/types.tsscript2.1 KB
scripts/lib/adapters/x/article.tsscript12.4 KB
scripts/lib/adapters/x/index.tsscript4.0 KB
scripts/lib/adapters/x/login.tsscript2.2 KB
scripts/lib/adapters/x/match.tsscript297 B
scripts/lib/adapters/x/payloads.tsscript1.6 KB
scripts/lib/adapters/x/session.tsscript1.4 KB
scripts/lib/adapters/x/shared.tsscript12.7 KB
scripts/lib/adapters/x/single.tsscript2.5 KB
scripts/lib/adapters/x/thread-loader.tsscript8.9 KB
scripts/lib/adapters/x/thread.tsscript9.0 KB
scripts/lib/adapters/x/types.tsscript600 B
scripts/lib/adapters/youtube/index.tsscript898 B
scripts/lib/adapters/youtube/transcript.tsscript13.0 KB
scripts/lib/adapters/youtube/utils.tsscript6.5 KB
scripts/lib/browser/cdp-client.tsscript6.7 KB
scripts/lib/browser/chrome-launcher.tsscript5.7 KB
scripts/lib/browser/cookie-sidecar.tsscript2.9 KB
scripts/lib/browser/interaction-gates.tsscript4.6 KB
scripts/lib/browser/network-journal.tsscript7.0 KB
scripts/lib/browser/page-snapshot.tsscript3.3 KB
scripts/lib/browser/profile.tsscript5.5 KB
scripts/lib/browser/session.tsscript4.5 KB
scripts/lib/cli.tsexecutablescript6.9 KB
scripts/lib/commands/convert.tsscript17.0 KB
scripts/lib/extract/document.tsscript846 B
scripts/lib/extract/html-cleaner.tsscript10.8 KB
scripts/lib/extract/html-extractor.tsscript2.2 KB
scripts/lib/extract/html-to-markdown.tsscript20.8 KB
scripts/lib/extract/markdown-renderer.tsscript4.6 KB
scripts/lib/media/default-downloader.tsscript5.1 KB
scripts/lib/media/markdown-media.tsscript13.3 KB
scripts/lib/media/media-utils.tsscript6.1 KB
scripts/lib/media/types.tsscript710 B
scripts/lib/types/defuddle-node.d.tsscript605 B
scripts/lib/types/shims.d.tsscript422 B
scripts/lib/utils/logger.tsscript645 B
scripts/lib/utils/url.tsscript300 B
scripts/package.jsonmanifest557 B

scripts/lib/browser/network-journal.ts

View on GitHub
import type { TargetSession } from "./cdp-client";
import type { Logger } from "../utils/logger";

type JsonObject = Record<string, unknown>;

export interface NetworkEntry {
  requestId: string;
  url: string;
  method: string;
  resourceType: string;
  timestamp: number;
  requestHeaders?: Record<string, string>;
  requestBody?: string;
  status?: number;
  statusText?: string;
  responseHeaders?: Record<string, string>;
  mimeType?: string;
  body?: string;
  bodyBase64?: boolean;
  bodyError?: string;
  failed?: boolean;
  failureReason?: string;
  finished: boolean;
}

function normalizeHeaders(headers: unknown): Record<string, string> | undefined {
  if (!headers || typeof headers !== "object") {
    return undefined;
  }
  return Object.fromEntries(
    Object.entries(headers as Record<string, unknown>).map(([key, value]) => [key, String(value)]),
  );
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

export class NetworkJournal {
  private readonly entries = new Map<string, NetworkEntry>();
  private lastActivityAt = Date.now();
  private started = false;

  constructor(
    private readonly session: TargetSession,
    private readonly log: Logger,
  ) {}

  async start(): Promise<void> {
    if (this.started) {
      return;
    }

    this.started = true;
    this.session.on("Network.requestWillBeSent", this.handleRequestWillBeSent);
    this.session.on("Network.responseReceived", this.handleResponseReceived);
    this.session.on("Network.loadingFinished", this.handleLoadingFinished);
    this.session.on("Network.loadingFailed", this.handleLoadingFailed);
    await this.session.send("Network.enable");
  }

  stop(): void {
    if (!this.started) {
      return;
    }
    this.session.off("Network.requestWillBeSent", this.handleRequestWillBeSent);
    this.session.off("Network.responseReceived", this.handleResponseReceived);
    this.session.off("Network.loadingFinished", this.handleLoadingFinished);
    this.session.off("Network.loadingFailed", this.handleLoadingFailed);
    this.started = false;
  }

  private touch(): void {
    this.lastActivityAt = Date.now();
  }

  private readonly handleRequestWillBeSent = (params: JsonObject): void => {
    const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
    const request = params.request as JsonObject | undefined;
    if (!requestId || !request) {
      return;
    }

    this.touch();
    this.entries.set(requestId, {
      requestId,
      url: String(request.url ?? ""),
      method: String(request.method ?? "GET"),
      resourceType: String(params.type ?? "Other"),
      timestamp: Date.now(),
      requestHeaders: normalizeHeaders(request.headers),
      requestBody: typeof request.postData === "string" ? request.postData : undefined,
      finished: false,
    });
  };

  private readonly handleResponseReceived = (params: JsonObject): void => {
    const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
    const response = params.response as JsonObject | undefined;
    if (!requestId || !response) {
      return;
    }

    this.touch();
    const existing = this.entries.get(requestId);
    if (!existing) {
      return;
    }

    existing.status = typeof response.status === "number" ? response.status : undefined;
    existing.statusText = typeof response.statusText === "string" ? response.statusText : undefined;
    existing.responseHeaders = normalizeHeaders(response.headers);
    existing.mimeType = typeof response.mimeType === "string" ? response.mimeType : undefined;
    this.entries.set(requestId, existing);
  };

  private readonly handleLoadingFinished = (params: JsonObject): void => {
    const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
    if (!requestId) {
      return;
    }

    this.touch();
    const existing = this.entries.get(requestId);
    if (!existing) {
      return;
    }
    existing.finished = true;
    this.entries.set(requestId, existing);
  };

  private readonly handleLoadingFailed = (params: JsonObject): void => {
    const requestId = typeof params.requestId === "string" ? params.requestId : undefined;
    if (!requestId) {
      return;
    }

    this.touch();
    const existing = this.entries.get(requestId);
    if (!existing) {
      return;
    }
    existing.finished = true;
    existing.failed = true;
    existing.failureReason = typeof params.errorText === "string" ? params.errorText : "Unknown error";
    this.entries.set(requestId, existing);
  };

  getEntries(): NetworkEntry[] {
    return Array.from(this.entries.values());
  }

  findEntries(predicate: (entry: NetworkEntry) => boolean): NetworkEntry[] {
    return this.getEntries().filter(predicate);
  }

  async waitForIdle(options: { idleMs?: number; timeoutMs?: number } = {}): Promise<void> {
    const idleMs = options.idleMs ?? 1_200;
    const timeoutMs = options.timeoutMs ?? 15_000;
    const startedAt = Date.now();

    while (Date.now() - startedAt < timeoutMs) {
      if (Date.now() - this.lastActivityAt >= idleMs) {
        return;
      }
      await sleep(Math.min(150, idleMs));
    }

    throw new Error("Timed out waiting for network idle");
  }

  async waitForResponse(
    predicate: (entry: NetworkEntry) => boolean,
    options: { timeoutMs?: number } = {},
  ): Promise<NetworkEntry> {
    const timeoutMs = options.timeoutMs ?? 10_000;
    const startedAt = Date.now();

    while (Date.now() - startedAt < timeoutMs) {
      const matched = this.getEntries().find((entry) => entry.finished && predicate(entry));
      if (matched) {
        return matched;
      }
      await sleep(150);
    }

    throw new Error("Timed out waiting for matching network response");
  }

  async ensureBody(entry: NetworkEntry): Promise<string | undefined> {
    if (entry.body !== undefined) {
      return entry.body;
    }
    if (entry.bodyError || entry.failed || !entry.finished) {
      return undefined;
    }

    try {
      const result = await this.session.send<{ body: string; base64Encoded: boolean }>("Network.getResponseBody", {
        requestId: entry.requestId,
      });
      entry.bodyBase64 = result.base64Encoded;
      entry.body = result.base64Encoded ? Buffer.from(result.body, "base64").toString("utf8") : result.body;
      return entry.body;
    } catch (error) {
      entry.bodyError = error instanceof Error ? error.message : String(error);
      this.log.debug(`Failed to fetch response body for ${entry.url}: ${entry.bodyError}`);
      return undefined;
    }
  }

  async getJsonBody(entry: NetworkEntry): Promise<unknown | null> {
    const body = await this.ensureBody(entry);
    if (!body) {
      return null;
    }

    try {
      return JSON.parse(body);
    } catch {
      return null;
    }
  }

  async toJSON(options: { includeBodies?: boolean } = {}): Promise<NetworkEntry[]> {
    const entries = this.getEntries();
    if (!options.includeBodies) {
      return entries;
    }

    await Promise.all(entries.map((entry) => this.ensureBody(entry)));
    return entries;
  }
}

slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall