aiskillstore/marketplace / skills/21pounder/web-scrape
web-scrape
VerifiedIntelligent web scraper with content extraction, multiple output formats, and error handling
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.8 KB |
| scripts/html_clean.js | script | 1.5 KB |
| skill-report.json | config | 43.9 KB |
scripts/html_clean.js
View on GitHubconst fs = require('fs');
const cheerio = require('cheerio');
const TurndownService = require('turndown');
const { resolve } = require('path');
const readStdin = () => {
return new Promise((resolve, reject) => {
let data = '';
process.stdin.setEncoding('utf8');
process.stdin.on('data', chunk => data += chunk);
process.stdin.on('end', () => resolve(data));
process.stdin.on('error', error => reject(error));
});
};
(async () => {
try {
const html = await readStdin();
if (!html) {
console.error('No input provided via stdin');
process.exit(1);
}
const $ = cheerio.load(html);
const noiseSelectors = [
'script', 'style', 'noscript', 'iframe', 'svg',
'nav', 'footer', 'header',
'.ad', '.ads', '.advertisement',
'#sidebar', '.sidebar',
'.comment', '.comments',
'[role="alert"]', '[role="banner"]', '[role="navigation"]'
];
$(noiseSelectors.join(',')).remove();
const turndownService = new TurndownService({
headingStyle: 'atx',
codeBlockStyle: 'fenced'
});
let contentHtml = $('article').html() || $('main').html() || $('body').html();
const markdown = turndownService.turndown(contentHtml || '');
console.log(markdown);
} catch (error) {
console.error('Error cleaning HTML:', error.message);
process.exit(1);
}
})();Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |