Skip to content

heymegabyte/agent-skills / 15-site-generation

site-generation

Unverified

End-to-end AI website generation pipeline. Claude Opus 4.8 emits Bolt-style <boltArtifact> envelopes (multi-file, plan-first) that customize Vite+React+Tailwind templates from pre-researched business data. Pre-research via APIs, media acquisition, brand extraction, visual inspection via GPT Image 2 vision, R2 upload (per-file content-type by extension), D1 status updates. Supports all business types: SaaS, portfolio, non-profit, restaurant, salon, medical, legal, retail, tech.

FileKindSize
SKILL.mdskill md11.7 KB
_real-ua.mjsscript1.4 KB
blog-import.mjsscript10.5 KB
bolt-artifact-protocol.mdmarkdown8.4 KB
build-breaking-rules.mdmarkdown86.0 KB
build-image-redirects.mjsscript3.4 KB
build-prompts.mdmarkdown86.3 KB
check-routes.shexecutablescript904 B
domain-features.mdmarkdown17.1 KB
enhance-blog-posts.mjsscript12.9 KB
homepage-block-library.mdmarkdown12.6 KB
local-seo.mdmarkdown7.6 KB
media-acquisition.mdmarkdown39.2 KB
non-technical-owner-onboarding.mdmarkdown8.3 KB
page-set-expansion.mdmarkdown19.1 KB
pseo-templates.mdmarkdown13.3 KB
quality-gates.mdmarkdown46.6 KB
research-pipeline.mdmarkdown20.5 KB
scan-assets.mjsscript3.0 KB
small-business-mode.mdmarkdown9.6 KB
source-fidelity-loop.mdmarkdown10.0 KB
squarespace-full-crawl.mjsscript2.8 KB
template-improvements-100.mdmarkdown15.0 KB
template-system.mdmarkdown39.7 KB
validate-assets.mjsscript5.0 KB
validate-banned-slop.mjsscript4.6 KB
validate-brand-colors.mjsexecutablescript6.1 KB
validate-citations.mjsscript6.0 KB
validate-color-contrast.mjsscript5.2 KB
validate-favicon-set.mjsscript4.9 KB
validate-html-entities.mjsscript3.4 KB
validate-image-relevance.mjsscript6.1 KB
validate-internal-links.mjsscript4.3 KB
validate-jsonld-schema.mjsscript5.1 KB
validate-nap-consistency.mjsexecutablescript8.0 KB
validate-page-count.mjsscript4.1 KB
validate-page-set-completeness.mjsscript9.3 KB
validate-photo-authenticity.mjsexecutablescript10.0 KB
validate-pwa-kit.mjsscript7.0 KB
validate-route-metadata.mjsscript7.3 KB
validate-source-fidelity.mjsscript6.0 KB
validate-typography.mjsscript4.0 KB
verticals/local-business-site.mdmarkdown3.8 KB
verticals/nonprofit-site.mdmarkdown3.3 KB
verticals/restaurant-site.mdmarkdown2.6 KB

blog-import.mjs

View on GitHub
#!/usr/bin/env node
// blog-import.mjs — promoted from njsk.org/scripts/clean-blog-corpus.mjs + enhance-blog-posts.mjs
// Reusable blog importer for any projectsites.dev rebuild with a source blog.
//
// Pipeline:
//   1. Crawl source blog (RSS preferred, JSON fallback, sitemap last)
//   2. Strip CSS/HTML residue (Squarespace fragments, raw <style>, .sqs-*, #block-*)
//   3. AI-restructure flat paragraphs → typed blocks (lead | heading | paragraph | quote | callout)
//   4. Generate per-post: SEO keywords (3-5), 120-180-char excerpt
//   5. pHash dedup hero images (sharp 8x8 DCT, hamming ≤6) — replaces md5
//   6. Emit src/data/blog-posts.ts (typed array) + public/images/blog/*
//
// Usage:
//   node blog-import.mjs --source https://example.com --out src/data/blog-posts.ts \
//                        --images-dir public/images/blog --delete-orphans
//   Env: OPENAI_API_KEY required for restructure pass.

import { writeFileSync, readFileSync, existsSync, mkdirSync, readdirSync, unlinkSync, statSync } from 'node:fs';
import { join, basename, extname } from 'node:path';
import { createHash } from 'node:crypto';

const args = Object.fromEntries(
  process.argv.slice(2).reduce((acc, cur, i, arr) => {
    if (cur.startsWith('--')) acc.push([cur.slice(2), arr[i + 1]?.startsWith('--') ? true : arr[i + 1]]);
    return acc;
  }, []),
);

const SOURCE_URL = args.source || process.env.SOURCE_URL;
const OUT_TS = args.out || 'src/data/blog-posts.ts';
const IMAGES_DIR = args['images-dir'] || 'public/images/blog';
const DELETE_ORPHANS = !!args['delete-orphans'];
const OPENAI_KEY = process.env.OPENAI_API_KEY;
import { REAL_UA_DESKTOP } from './_real-ua.mjs';
const REAL_UA = process.env.REAL_UA || REAL_UA_DESKTOP;

if (!SOURCE_URL) {
  console.error('Usage: node blog-import.mjs --source https://... --out src/data/blog-posts.ts --images-dir public/images/blog [--delete-orphans]');
  process.exit(1);
}
if (!OPENAI_KEY) {
  console.error('OPENAI_API_KEY required for AI restructure pass');
  process.exit(1);
}

const FETCH_HEADERS = {
  'User-Agent': REAL_UA,
  Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept-Encoding': 'gzip, deflate, br',
  'Sec-Fetch-Site': 'none',
  'Sec-Fetch-Mode': 'navigate',
  'Sec-Fetch-User': '?1',
  'Sec-Fetch-Dest': 'document',
  'Upgrade-Insecure-Requests': '1',
};

mkdirSync(IMAGES_DIR, { recursive: true });

// 1. Crawl — RSS first
async function fetchRss() {
  const candidates = ['/blog?format=rss', '/feed.xml', '/rss.xml', '/feed', '/atom.xml'];
  for (const path of candidates) {
    try {
      const r = await fetch(new URL(path, SOURCE_URL), { headers: FETCH_HEADERS });
      if (!r.ok) continue;
      const xml = await r.text();
      if (xml.includes('<item') || xml.includes('<entry')) return xml;
    } catch {}
  }
  return null;
}
async function fetchSquarespaceJson() {
  try {
    const r = await fetch(new URL('/blog?format=json-pretty', SOURCE_URL), { headers: FETCH_HEADERS });
    if (!r.ok) return null;
    return await r.json();
  } catch {
    return null;
  }
}

console.log(`[blog-import] crawling ${SOURCE_URL}…`);
const rss = await fetchRss();
const sqs = !rss ? await fetchSquarespaceJson() : null;
if (!rss && !sqs) {
  console.error('[blog-import] no RSS feed or Squarespace JSON found — falling back to sitemap+scrape not yet implemented');
  process.exit(1);
}

// Parse posts
function parseRssItems(xml) {
  const items = [];
  const itemRegex = /<item>([\s\S]*?)<\/item>/g;
  let m;
  while ((m = itemRegex.exec(xml)) !== null) {
    const block = m[1];
    const get = (tag) => {
      const r = new RegExp(`<${tag}[^>]*>([\\s\\S]*?)</${tag}>`).exec(block);
      return r ? r[1].replace(/<!\[CDATA\[|\]\]>/g, '').trim() : '';
    };
    items.push({
      title: get('title'),
      link: get('link'),
      pubDate: get('pubDate'),
      content: get('content:encoded') || get('description'),
      author: get('dc:creator') || get('author'),
    });
  }
  return items;
}

const rawPosts = rss
  ? parseRssItems(rss)
  : sqs.items?.map((i) => ({
      title: i.title,
      link: `${SOURCE_URL}${i.fullUrl}`,
      pubDate: new Date(i.publishOn || i.addedOn).toISOString(),
      content: i.body,
      author: i.author?.displayName || '',
    })) || [];

console.log(`[blog-import] ${rawPosts.length} posts parsed`);

// 2. Strip CSS/HTML residue
function stripResidue(html) {
  return html
    .replace(/<style[^>]*>[\s\S]*?<\/style>/gi, '')
    .replace(/<script[^>]*>[\s\S]*?<\/script>/gi, '')
    .replace(/id="block-yui_[^"]*"/gi, '')
    .replace(/class="sqs-[^"]*"/gi, '')
    .replace(/class="margin-wrapper[^"]*"/gi, '')
    .replace(/class="content-wrapper[^"]*"/gi, '')
    .replace(/data-block-type="\d+"/gi, '')
    .replace(/<div[^>]*>\s*<\/div>/gi, '')
    .replace(/<p[^>]*>\s*<\/p>/gi, '')
    .replace(/\n{3,}/g, '\n\n')
    .trim();
}

// 3. AI restructure → typed blocks
async function restructure(title, html) {
  const cleaned = stripResidue(html);
  // Strip remaining tags except <img>/<a>/<blockquote> for context
  const text = cleaned.replace(/<\/?(?:p|div|span|br)\b[^>]*>/gi, '\n').replace(/<[^>]+>/g, '').replace(/\n+/g, '\n').trim();
  const r = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${OPENAI_KEY}` },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      temperature: 0.3,
      response_format: { type: 'json_object' },
      messages: [
        {
          role: 'system',
          content: `You restructure raw blog text into typed visual blocks. Preserve direct quotes verbatim. Never alter names, dates, or facts.

Block types:
- lead: opening paragraph (one per post, drop-cap styled)
- heading: section header (level 2)
- paragraph: body paragraph
- quote: direct quotation with attribution
- callout: emphasized aside / scripture / pullquote

Return JSON: { excerpt: string (120-180 chars), keywords: string[] (3-5), blocks: [{type, text, level?}] }
Split long paragraphs into multiple paragraph blocks. Each block stands alone visually.`,
        },
        { role: 'user', content: `Title: ${title}\n\nContent:\n${text.slice(0, 8000)}` },
      ],
    }),
  });
  if (!r.ok) {
    console.error(`[blog-import] OpenAI fail for "${title}": ${r.status}`);
    return null;
  }
  const data = await r.json();
  return JSON.parse(data.choices[0].message.content);
}

// 5. pHash dedup (8x8 DCT via sharp)
async function phash(buffer) {
  try {
    const sharp = (await import('sharp')).default;
    const raw = await sharp(buffer).greyscale().resize(32, 32, { fit: 'fill' }).raw().toBuffer();
    // 32x32 grayscale → DCT → top-left 8x8 → median threshold → 64-bit hash
    const dct = computeDct(raw, 32);
    const block = [];
    for (let y = 0; y < 8; y++) for (let x = 0; x < 8; x++) block.push(dct[y * 32 + x]);
    const median = [...block].sort((a, b) => a - b)[32];
    return block.map((v) => (v > median ? '1' : '0')).join('');
  } catch {
    // sharp not installed → fallback md5
    return createHash('md5').update(buffer).digest('hex');
  }
}
function computeDct(pixels, size) {
  const out = new Float64Array(size * size);
  for (let v = 0; v < size; v++) {
    for (let u = 0; u < size; u++) {
      let sum = 0;
      for (let y = 0; y < size; y++) {
        for (let x = 0; x < size; x++) {
          sum += pixels[y * size + x] * Math.cos(((2 * x + 1) * u * Math.PI) / (2 * size)) * Math.cos(((2 * y + 1) * v * Math.PI) / (2 * size));
        }
      }
      const cu = u === 0 ? 1 / Math.sqrt(2) : 1;
      const cv = v === 0 ? 1 / Math.sqrt(2) : 1;
      out[v * size + u] = (cu * cv * sum) / 4;
    }
  }
  return out;
}
function hamming(a, b) {
  let d = 0;
  for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) d++;
  return d;
}

// Process posts
const slugify = (s) => s.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '').slice(0, 80);
const seenHashes = [];
const usedImages = new Set();
const finalPosts = [];

for (const raw of rawPosts) {
  const slug = slugify(raw.title);
  if (!slug) continue;

  // Hero image
  const imgMatch = /<img[^>]+src=["']([^"']+)["']/i.exec(raw.content);
  let hero = null;
  if (imgMatch) {
    const imgUrl = imgMatch[1];
    try {
      const r = await fetch(imgUrl, { headers: FETCH_HEADERS });
      const buf = Buffer.from(await r.arrayBuffer());
      const hash = await phash(buf);
      const dup = seenHashes.find((h) => hamming(h.hash, hash) <= 6);
      if (dup) {
        hero = dup.path;
      } else {
        const ext = extname(new URL(imgUrl).pathname).toLowerCase() || '.jpg';
        const fname = `${slug}${ext}`;
        const dest = join(IMAGES_DIR, fname);
        writeFileSync(dest, buf);
        hero = `/images/blog/${fname}`;
        seenHashes.push({ hash, path: hero });
        usedImages.add(fname);
      }
    } catch (e) {
      console.warn(`[blog-import] hero fetch fail for ${slug}: ${e.message}`);
    }
  }

  const restructured = await restructure(raw.title, raw.content);
  if (!restructured) continue;

  finalPosts.push({
    slug,
    title: raw.title,
    date: new Date(raw.pubDate || Date.now()).toISOString().slice(0, 10),
    author: raw.author || '',
    excerpt: restructured.excerpt,
    keywords: restructured.keywords,
    hero,
    blocks: restructured.blocks,
  });
  console.log(`[blog-import] ✓ ${slug} (${restructured.blocks.length} blocks)`);
}

// 6. Emit TS file
const ts = `// AUTO-GENERATED by ~/.agentskills/15-site-generation/blog-import.mjs
// Source: ${SOURCE_URL}
// Generated: ${new Date().toISOString()}
// Posts: ${finalPosts.length}

export interface BlogBlock {
  type: 'lead' | 'heading' | 'paragraph' | 'quote' | 'callout';
  text: string;
  level?: number;
}
export interface BlogPost {
  slug: string;
  title: string;
  date: string;
  author: string;
  excerpt: string;
  keywords: string[];
  hero: string | null;
  blocks: BlogBlock[];
}

export const blogPosts: BlogPost[] = ${JSON.stringify(finalPosts, null, 2)};
`;
mkdirSync(OUT_TS.split('/').slice(0, -1).join('/') || '.', { recursive: true });
writeFileSync(OUT_TS, ts);
console.log(`[blog-import] wrote ${finalPosts.length} posts → ${OUT_TS}`);

// Delete orphans
if (DELETE_ORPHANS) {
  let deleted = 0;
  for (const f of readdirSync(IMAGES_DIR)) {
    if (!usedImages.has(f) && !seenHashes.some((h) => h.path.endsWith(f))) {
      unlinkSync(join(IMAGES_DIR, f));
      deleted++;
    }
  }
  console.log(`[blog-import] deleted ${deleted} orphan images from ${IMAGES_DIR}`);
}

console.log(`[blog-import] DONE — ${finalPosts.length} posts, ${seenHashes.length} unique hero images`);
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall