oxylabs/agent-skills / skills/agent-browser
agent-browser
VerifiedConnects to Oxylabs remote agent browsers over the Chrome DevTools Protocol (CDP) with Playwright or Puppeteer. Managed browser environment, residential proxies, geo-targeting, persistent sessions and profiles, session recording and live VNC inspection for debugging. Use instead of WebFetch or a local browser whenever a site renders with JavaScript, serves access challenges, verification interstitials or bot-management block pages, or needs a full browser session, screenshots or PDFs. Covers connection, retries, error recovery, built-in challenge handling (the page_handler parameter, the oxylabs-page-status page state reported by the page handler, manual triggering), telling a handled challenge from a block that needs a fresh identity, and reliable access to publicly available data without manual intervention.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 16.1 KB |
| challenges.md | markdown | 17.0 KB |
| errors.md | markdown | 9.1 KB |
| examples.md | markdown | 11.5 KB |
| parameters.md | markdown | 8.0 KB |
| scripts/playwright_scrape.js | script | 7.1 KB |
| scripts/playwright_scrape.py | script | 6.3 KB |
| targets.md | markdown | 12.5 KB |
examples.md
View on GitHub# Examples
All examples read `OXY_AB_USERNAME` / `OXY_AB_PASSWORD` and connect to `wss://USERNAME:PASSWORD@hb.oxylabs.io`. The `endpointUrl`, `connectWithBackoff`,
`blockHeavyResources`, `watchOxylabsErrors`, `watchPageStatus`, `waitForPageStatus` and `scrape` helpers are the ones from
`scripts/playwright_scrape.js`. Challenge handling snippets (page status listeners for every client, manual trigger,
retry on error, profile setup with a challenge) are in `challenges.md`.
## Puppeteer
```javascript
const puppeteer = require("puppeteer");
const USERNAME = process.env.OXY_AB_USERNAME;
const PASSWORD = process.env.OXY_AB_PASSWORD;
// Same policy as scripts/playwright_scrape.js: Puppeteer reports "Unexpected server response: <status>" without the body.
const RETRYABLE =
/\b(429|50[0234]) [A-Z]|response: (429|50[0234])\b|CDP_SESSION_IN_USE|CDP_NO_BROWSERS_AVAILABLE|CDP_BROWSER_OVERWORKED|CDP_BAD_PROXY|CDP_GENERAL_ERROR|[Tt]imeout|ECONNRESET/;
async function connect(params, attempts = 6) {
const q = new URLSearchParams(params).toString();
const browserWSEndpoint = `wss://${USERNAME}:${PASSWORD}@hb.oxylabs.io${q ? "?" + q : ""}`;
for (let i = 0; ; i++) {
try {
return await puppeteer.connect({ browserWSEndpoint, protocolTimeout: 60000 });
} catch (err) {
if (i >= attempts - 1 || !RETRYABLE.test(err.message)) throw err;
await new Promise((r) => setTimeout(r, Math.min(60000, 1000 * 2 ** i) * (0.5 + Math.random())));
}
}
}
(async () => {
// Most jobs need only cc. Blocked after a fresh identity with the right geo? Next rung (targets.md section 4):
// connect({ cc: "US", browser: "oxybrowser", fp_tag: "oxyfp_mac_optimized" })
const browser = await connect({ cc: "US" });
try {
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on("request", (req) =>
["image", "stylesheet", "media", "font"].includes(req.resourceType()) ? req.abort() : req.continue());
page.on("response", (resp) => {
const d = resp.headers()["x-error-description"];
if (d) console.warn("Oxylabs-side error", resp.status(), resp.url(), d);
});
await page.goto("https://example.com", { waitUntil: "domcontentloaded", timeout: 30000 });
console.log(await page.title());
} finally {
await browser.close();
}
})();
```
## Playwright (Python, async)
```python
import asyncio, os, random, re
from playwright.async_api import async_playwright
USERNAME = os.environ["OXY_AB_USERNAME"]
PASSWORD = os.environ["OXY_AB_PASSWORD"]
RETRYABLE = re.compile( # same policy as scripts/playwright_scrape.py
r"\b(429|50[0234]) [A-Z]|response: (429|50[0234])\b|CDP_SESSION_IN_USE|CDP_NO_BROWSERS_AVAILABLE|"
r"CDP_BROWSER_OVERWORKED|CDP_BAD_PROXY|CDP_GENERAL_ERROR|[Tt]imeout|ECONNRESET")
async def connect(pw, params, attempts=6):
q = "&".join(f"{k}={v}" for k, v in params.items())
url = f"wss://{USERNAME}:{PASSWORD}@hb.oxylabs.io" + (f"?{q}" if q else "")
for i in range(attempts):
try:
return await pw.chromium.connect_over_cdp(url, timeout=60_000)
except Exception as err: # noqa: BLE001
if i == attempts - 1 or not RETRYABLE.search(str(err)):
raise
await asyncio.sleep(min(60, 2 ** i) * (0.5 + random.random()))
async def main():
async with async_playwright() as pw:
browser = await connect(pw, {"cc": "US"})
try:
context = browser.contexts[0] if browser.contexts else await browser.new_context()
page = await context.new_page()
await page.route("**/*", lambda r: r.abort()
if r.request.resource_type in {"image", "stylesheet", "media", "font"}
else r.continue_())
page.on("response", lambda r: r.headers.get("x-error-description")
and print("Oxylabs-side error", r.status, r.url, r.headers["x-error-description"]))
await page.goto("https://example.com", wait_until="domcontentloaded", timeout=30_000)
await page.screenshot(path="page.png", full_page=True)
print(await page.title())
finally:
await browser.close()
asyncio.run(main())
```
## Raw CDP over WebSocket (no library)
Node's built-in `WebSocket` (and many minimal clients) silently drop `user:pass@` from the URL, which the
service reports as `401 missing auth header`. Build the header yourself. The service decodes with the
URL-safe base64 alphabet and requires exactly one `:` in the decoded value.
```javascript
const USERNAME = process.env.OXY_AB_USERNAME;
const PASSWORD = process.env.OXY_AB_PASSWORD;
if (PASSWORD.includes(":")) throw new Error("password may not contain ':' for Basic auth");
const token = Buffer.from(`${USERNAME}:${PASSWORD}`).toString("base64");
if (/[+/]/.test(token)) console.warn("base64 contains + or /; use base64url encoding: " + Buffer.from(`${USERNAME}:${PASSWORD}`).toString("base64url"));
const ws = new WebSocket("wss://hb.oxylabs.io/?cc=US", { headers: { Authorization: `Basic ${token}` } });
let nextId = 1;
const pending = new Map();
const send = (method, params = {}, sessionId) =>
new Promise((resolve, reject) => {
const id = nextId++;
pending.set(id, { resolve, reject });
ws.send(JSON.stringify({ id, method, params, ...(sessionId && { sessionId }) }));
});
ws.addEventListener("message", (ev) => {
const msg = JSON.parse(ev.data);
if (msg.id && pending.has(msg.id)) {
const { resolve, reject } = pending.get(msg.id);
pending.delete(msg.id);
msg.error ? reject(new Error(`${msg.error.code} ${msg.error.message}`)) : resolve(msg.result);
}
});
ws.addEventListener("close", (ev) => console.log("closed", ev.code, ev.reason)); // 3000 + CDP_* on failure
ws.addEventListener("open", async () => {
try {
const { value: sessionId } = await send("__session_id");
console.log("session_id:", sessionId, "inspect: https://hb.oxylabs.io/novnc/?id=" + sessionId);
const { targetId } = await send("Target.createTarget", { url: "about:blank" });
const { sessionId: cdpSession } = await send("Target.attachToTarget", { targetId, flatten: true });
await send("Page.enable", {}, cdpSession);
await send("Page.navigate", { url: "https://example.com" }, cdpSession);
} finally {
setTimeout(() => ws.close(1000), 5000); // always close: the session holds a concurrency slot
}
});
```
## Resume a named session
`session_name` alone keeps the remote browser alive for 10 minutes after you disconnect. What survives
depends on **how** you disconnect:
| Disconnect method | Session resumable | Open pages and their cookies kept |
|-------------------|-------------------|-----------------------------------|
| Puppeteer `browser.disconnect()` | yes | yes |
| Process exit / dropped connection | yes | yes |
| Playwright or Puppeteer `browser.close()` | yes | **no** (pages closed, cookies cleared) |
Puppeteer hand-over (process A):
```javascript
const puppeteer = require("puppeteer-core");
const browser = await puppeteer.connect({ browserWSEndpoint: endpointUrl({ session_name: "job-42", cc: "US" }) });
const page = await browser.newPage();
await page.goto("https://example.com/");
// ... choose region and currency, accept the cookie banner ...
await browser.disconnect(); // NOT close(): keeps the page and cookies for the next connection
```
Resume (process B, within 10 minutes, same credentials; Playwright or Puppeteer):
```javascript
const browser = await connectWithBackoff({ session_name: "job-42", cc: "US" }); // 429 CDP_SESSION_IN_USE while A is still attached
try {
const ctx = browser.contexts()[0];
const page = ctx.pages().find((p) => p.url().includes("example.com")) ?? (await ctx.newPage());
await page.goto("https://example.com/catalog"); // region and currency choice still applied
} finally {
await browser.close(); // last user of the session: close for real
}
```
If every process uses Playwright, keep the first connection open until the process exits instead of calling
`close()`. For state that must outlive the 10-minute window use a profile (next example). To end a named
session early, connect once more with `keep_alive=false` and close.
## Profile setup and consumer runs
Most jobs need no profile: every session is a fresh browser configuration on a fresh IP. Use one when cookies must
survive between jobs (the cookie a site sets once its challenge completes, cookie consent, site preferences). Then
split the work in two roles. **Setup** runs once per profile and is
the only connection that sends `profile_save=true`. **Consumers** send `profile_name` alone: read-only, no write lock,
so they can never hit `409` and can start while others run. In production these are two services: one prepares and
validates profiles, the other only uses them.
```javascript
const identity = {
profile_name: "acme-us-01", cc: "US", device: "desktop",
proxy_resi_ses_id: "acmeus01", proxy_resi_ses_time: "30",
};
// Setup service: run once per profile name. Let the site set its cookies, verify, close (closing writes the profile).
async function setupProfile(entryUrl) {
const browser = await connectWithBackoff({ ...identity, profile_save: "true" });
try {
const page = await browser.contexts()[0].newPage();
const response = await page.goto(entryUrl, { waitUntil: "domcontentloaded" });
await page.waitForTimeout(5000); // let late JS and cookies settle
if ((await classify(page, response)) !== "ok") throw new Error("setup failed: block page, pick a new profile name"); // classify() from targets.md
// ... accept cookie consent or set site preferences here if the target needs it ...
} finally {
await browser.close();
}
}
// Consumer service: any number of runs, never profile_save.
await setupProfile("https://target.example/");
await scrape("https://target.example/listing/1", identity);
await scrape("https://target.example/listing/2", identity);
```
Use `browser.contexts()[0]`; a `newContext()` is not backed by the profile. Keep every geo/device value identical
across setup and consumers. When consumers start seeing block pages, do not re-save the profile from a consumer: run
`setupProfile` again under a new profile name and a new sticky id.
## Session id, live inspection and recording
```javascript
const browser = await connectWithBackoff({ cc: "US", record: "true", record_name: "job-42" });
try {
const context = browser.contexts()[0];
const page = await context.newPage();
const cdp = await context.newCDPSession(page);
const { value: sessionId } = await cdp.send("__session_id");
console.log(`session_id: ${sessionId}`);
console.log(`live view: https://hb.oxylabs.io/novnc/?id=${sessionId}`);
await page.goto("https://example.com");
} finally {
await browser.close(); // the recording is finalised and appears in https://hb.oxylabs.io/dashboard
}
```
## Fan-out with launch pacing
```javascript
async function mapWithPacing(urls, params, concurrency = 8, launchGapMs = 150) {
const results = new Array(urls.length);
let next = 0;
async function worker() {
while (next < urls.length) {
const i = next++;
await new Promise((r) => setTimeout(r, launchGapMs)); // stay well under 10 launches/second
try {
results[i] = await scrape(urls[i], params);
} catch (err) {
results[i] = { error: err.message };
}
}
}
await Promise.all(Array.from({ length: concurrency }, worker));
return results;
}
```
Each `scrape` call opens and closes its own session, so `concurrency` is the number of concurrent sessions
(default account cap 100).
*This content is not intended to provide legal advice, and scraping activities may be subject to applicable laws and regulations.*
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |