aboylabs/ai-agent-skills / skills/research/web-scraping
web-scraping
VerifiedExtract data from websites, including JavaScript-rendered SPAs and dynamic content
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.5 KB |
| references/browser-use-cloud-api.md | markdown | 5.3 KB |
| references/browser-use-cloud.md | markdown | 3.5 KB |
| references/browserbase-api.md | markdown | 7.6 KB |
| references/cloudflare-bypass-strategies.md | markdown | 6.4 KB |
| references/cloudflare-turnstile-bypass.md | markdown | 4.5 KB |
| references/nextjs-ssr-scraping.md | markdown | 2.6 KB |
| references/race-condition-order-matching.md | markdown | 9.2 KB |
| references/security-audit-pattern.md | markdown | 7.1 KB |
| references/security-audit-workflow.md | markdown | 6.8 KB |
| references/spa-api-discovery.md | markdown | 5.5 KB |
| references/tinyfish-js-reverse-engineering.md | markdown | 6.7 KB |
| references/trendshift-analysis.md | markdown | 2.2 KB |
references/security-audit-pattern.md
View on GitHub# Web Application Security Audit Pattern
Structured approach for security auditing web applications for responsible disclosure.
## Use Case
User wants to find vulnerabilities in a web application to report to the owner (responsible disclosure, bug bounty, security research).
## Audit Structure
### 1. API Endpoint Discovery & Testing
Test common endpoints for:
- **Exposure:** Admin/config/env endpoints accessible without auth
- **SQL Injection:** `?id=1' OR '1'='1`
- **Path Traversal:** `/api/../../../etc/passwd`
- **IDOR:** `/api/user/1`, `/api/order/1` (access other users' data)
- **Authentication bypass:** POST without auth token
**Pattern:**
```python
endpoints = [
# Public endpoints (should work)
("/api/listings", "GET", None),
# Admin/sensitive (should be protected)
("/api/admin", "GET", None),
("/api/config", "GET", None),
("/api/env", "GET", None),
# SQL injection tests
("/api/listings?id=1' OR '1'='1", "GET", None),
# Path traversal
("/api/../../../etc/passwd", "GET", None),
# IDOR
("/api/user/1", "GET", None),
# Auth bypass
("/api/listings", "POST", {"data": "test"}),
]
for path, method, data in endpoints:
try:
url = f"{BASE_URL}{path}"
if method == "GET":
r = requests.get(url, timeout=5)
elif method == "POST":
r = requests.post(url, json=data, timeout=5)
results.append({
"endpoint": path,
"status": r.status_code,
"size": len(r.content),
"note": classify_response(r)
})
except requests.exceptions.Timeout:
results.append({
"endpoint": path,
"status": "TIMEOUT",
"note": "Possible DoS vector"
})
```
### 2. Authentication Testing
Test if endpoints require authentication:
```python
tests = [
("No auth header", {}),
("Invalid token", {"Authorization": "Bearer invalid"}),
("SQL injection in token", {"Authorization": "Bearer ' OR '1'='1"}),
("Empty token", {"Authorization": "Bearer "}),
]
for test_name, headers in tests:
r = requests.get(f"{BASE_URL}/api/listings", headers=headers)
authenticated = r.status_code != 401
```
### 3. Rate Limiting Test
Send rapid requests to detect rate limiting:
```python
for i in range(50):
r = requests.get(f"{BASE_URL}/api/listings", timeout=2)
if r.status_code == 429:
print(f"Rate limited after {i+1} requests")
break
else:
print("⚠️ No rate limiting detected - DoS vulnerability")
```
### 4. CORS Configuration
Check for permissive CORS:
```python
r = requests.options(f"{BASE_URL}/api/listings", headers={
"Origin": "https://evil.com",
"Access-Control-Request-Method": "POST"
})
if r.headers.get("Access-Control-Allow-Origin") == "*":
print("⚠️ Permissive CORS - allows any origin")
```
### 5. Input Validation
Test with malicious payloads:
```python
payloads = [
{"amount": -1000, "price": 1}, # Negative values
{"amount": 999999999999, "price": 0.000001}, # Overflow
{"amount": "'; DROP TABLE listings; --", "price": 1}, # SQL injection
{"amount": "<script>alert('xss')</script>", "price": 1}, # XSS
]
for payload in payloads:
r = requests.post(f"{BASE_URL}/api/listings", json=payload)
if r.status_code in [200, 201]:
print(f"⚠️ Invalid input accepted: {payload}")
```
### 6. Data Analysis
Analyze existing data for anomalies:
```python
listings = requests.get(f"{BASE_URL}/api/listings").json()
for listing in listings:
# Check for negative values (should be impossible)
if listing.get("amount", 0) < 0 or listing.get("price", 0) < 0:
print(f"🚨 CRITICAL: Negative values in listing {listing['id']}")
# Check for price manipulation
if listing["amount"] > 0:
price_per_unit = listing["price"] / listing["amount"]
if price_per_unit < 0.00001:
print(f"⚠️ Suspiciously low price: {listing['id']}")
if price_per_unit > 100:
print(f"⚠️ Suspiciously high price: {listing['id']}")
```
## Report Structure
### Executive Summary
- Date, target, purpose
- Number of vulnerabilities found
- Severity breakdown
### Findings by Category
1. **API Endpoint Testing**
- Status codes, response sizes
- Classify: ✅ OK, ⚠️ WARNING, 🚨 ERROR
2. **Authentication**
- Which endpoints require auth
- Bypass attempts
3. **Rate Limiting**
- Presence/absence
- Threshold if detected
4. **CORS**
- Configuration
- Permissiveness level
5. **Input Validation**
- Accepted/rejected payloads
- Sanitization effectiveness
6. **Data Integrity**
- Anomalies in existing data
- Business logic flaws
### Recommendations
Prioritized by severity:
- 🚨 **CRITICAL:** Immediate action required (DoS, data breach risk)
- ⚠️ **MEDIUM:** Should fix soon (CORS, missing validation)
- ℹ️ **LOW:** Nice to have (data quality, UX issues)
### Positive Findings
List what's done right (builds trust with recipient).
### Disclosure Timeline
- Discovery date
- Disclosure date
- Requested response time (7 days)
- Public disclosure timeline (30 days after fix)
## Case Study: rpow2swap.com (May 2026)
**Findings:**
1. 🚨 **CRITICAL:** No rate limiting (50+ requests without throttling)
2. ⚠️ **MEDIUM:** Permissive CORS (`Access-Control-Allow-Origin: *`)
3. ⚠️ **MEDIUM:** Multiple endpoint timeouts (possible DoS vector)
4. ℹ️ **LOW:** 2 suspicious listings with abnormally high prices
**Positive:**
- ✅ POST endpoints protected (401 auth)
- ✅ SQL injection attempts blocked
- ✅ Path traversal handled properly
**Contact:** rpow2swap@protonmail.com (found in minified JS)
**Files:**
- `/tmp/rpow2_security_audit.py` — Automated audit script
- `/tmp/rpow2_security_audit.md` — Full report
- `/tmp/disclosure_email.txt` — Email template
## Ethical Guidelines
**DO:**
- Test read-only operations
- Use test accounts when possible
- Report findings responsibly
- Give reasonable response time (7-30 days)
- Offer to help with remediation
**DON'T:**
- Exploit vulnerabilities for personal gain
- Access/modify other users' data
- Perform destructive tests on production
- Publicly disclose before fix (unless critical + unresponsive)
- Demand payment (unless bug bounty program)
## Pitfalls
- **Mainnet vs Testnet:** Always verify you're testing the right environment. Check wallet addresses on blockchain explorers.
- **False positives:** Timeouts may be network issues, not vulnerabilities.
- **Legal risk:** Ensure you have permission or are within legal safe harbor (responsible disclosure laws vary by country).
- **Scope creep:** Focus on security issues, not feature requests or UX complaints.
- **Tone:** Keep report professional and constructive, not accusatory.
## Tools
- `requests` — HTTP client
- `jq` — JSON parsing
- Browser DevTools — Network inspection
- Burp Suite / OWASP ZAP — Advanced testing (optional)
## References
- OWASP Top 10: https://owasp.org/www-project-top-ten/
- HackerOne Disclosure Guidelines: https://www.hackerone.com/disclosure-guidelines
- Bug Bounty Platforms: HackerOne, Bugcrowd, Intigriti
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |