Skip to content

aranculleton/last-job-search / .claude/skills/job-scraper

scrape

Verified

Finds new job postings matching your profile via installed portal-search CLIs (LinkedIn, freehire, local job boards, and any skills added with /add-portal; whichever are on, per tools/portals.py) and employer ATS boards. Deduplicates across runs. Triggers on: job scrape, find jobs, search jobs, new jobs, job search, scrape jobs, /scrape

FileKindSize
SKILL.mdskill md8.9 KB
manual-round.mdmarkdown39.5 KB
rules.mdmarkdown19.8 KB
# Why a round does what it does

The reasons behind the job-scraper skill's rules, and where the code keeps each. A chat running
a search needs none of this: the skill's card (`SKILL.md`) is what it reads. Read this when
changing the round, or when the person asks why a job was cut, kept or ranked where it is.

**How the round scores.** `tools/run_round.py` spends no model tokens: its stages are rule
counts and keyword reads, and its log says so at each one ("0 model tokens"). The order is
fit times opportunity over 100 (`tools/fit.py`, `tools/opportunity.py`); the score shown is the
fit, 0-100. The tiered path below (a mechanical prefilter, a small-model knockout, full scoring
one posting per agent, adversarial verification over 70) is the hand-run round in
`manual-round.md`, used only when `run_round.py` cannot run.

## The rules every search keeps, and why

`/scrape` is a ranking tool, not a link collector: every posting presented is scored, in
score order, and a posting that fails a gate is excluded and recorded, never ranked.

- Rank down, do not cut. A posting is a wish list. It names ten tools and the team uses
  six, and the missing one is almost never what decides the hire. **One missing skill is
  never a reason to drop a role**, and neither is a year of experience, a preferred degree,
  or a posted range under target (bar the one salary cut below). Those are scoring inputs; the
  role still surfaces and the candidate decides. Enforced in `tools/fit.py`, which weights every
  gap by how hard they asked and by what sits next to it, and in `tools/text_screen.py`, where a
  degree minimum and a range above the cut line are flags rather than cuts.
- Only these are hard cuts, because no framing survives them, and each one carries the
  quoted sentence it was cut on: an **eligibility** bar the profile cannot clear (a citizenship,
  residency or clearance requirement, or a requirement to live elsewhere), a **language**
  requirement (or a title written in another language), a **location** the Location table fails,
  a **professional credential** gate (actuarial exams, an accounting or engineering licence), a
  role whose **core is a discipline the person does not hold** that the profile names as a
  different job (`search.py` `RESEARCH_ML`, research machine learning for an analyst; nothing for
  a researcher), a vacancy **open to the employer's own staff only** ("internal applicants only",
  an "Internal Mobility" title), and a stated salary range whose **top** sits below the person's
  cut line (`SALARY_CUT_BELOW` in `profile/search.py`, well under the walk-away floor: a different
  job level, not a negotiation). Those are different jobs, not missing skills. Everything else
  gets ranked. A top between the cut line and the floor is a flag, never a cut, in
  `tools/text_screen.py` and in `tools/score_posting.py`, which reads extracted fields.
- **Every posting ruled out is written back to `seen_jobs.json`** with `tools/exclude.py add`,
  which stores `status: "excluded"`, a checkable `exclusion_reason`, a date and what the ruling
  rests on, so it never resurfaces while that holds. A ruling under the person's screening
  rules lapses when the part it rests on changes (`--rests-on "triage #4"`, or a reason citing
  "blocker 4"; with neither, any change to the rules) and the posting is screened again when
  next found; one in the person's own words holds until `exclude.py lift`. **An exclusion stands
  only while today's rules would still cut it** (`throttle.is_stale_exclusion`): one resting on a
  named part of the person's rules stands while that part does, and any other is judged by its
  reason. Eligibility, a language, a credential, a different discipline, internal-only and a fact
  about the posting (closed, a duplicate) stand; a pay reason stands only on a range topping out
  below `SALARY_CUT_BELOW`; a place only where the location gate still fails the place the entry
  recorded; a degree, a score or a years bar never. A round records the text screen's own cuts
  this way by itself (`tools/text_screen.py --seen`, which `run_round.py` passes), each with its
  quote, so the next round drops them before hydrate fetches them again. The prefilter's drops
  (age, place, title) are not recorded: they are decided again for free, before any fetch, and
  thousands a round would bury the rulings. The gate that enforces
  that is `throttle.ruled_out()`, and it runs at **prefilter**, before `hydrate.py` spends an
  HTTP call fetching the body: a large share of everything reaching hydrate is a posting already
  ruled out. `prefilter.py` imports the throttle's own function rather than reimplementing it, so
  the shortlist is unchanged; `run_round.py --no-seen-read` turns it off, which is what you
  want after changing the scoring rules and wanting everything previously excluded
  re-evaluated. Exclusions are per
  posting, never per company: a different role at the same employer stays in scope.
- **`job_scraper/do_not_target.json` is the one company-level exception, and it is a
  different claim.** A posting exclusion says a role was rejected. An entry there says the
  employer has **no relevant presence for that function in that city**, so there is no role
  to stay in scope: an office that houses only sales, or only production, has no role to stay
  in scope for unless one of those is the person's function. `tools/throttle.py` reads it and
  skips the employer with
  the reason. Every entry names who established it and carries a **`review_after` date**,
  after which it stops suppressing and is reported as expired, because a company standing up
  a team in that city is exactly the fact that goes stale. Add an entry only on positive
  evidence of absence, never as a shortcut for "this one was a bad fit".
- Every posting presented, and every posting documents are drafted for, carries a clickable
  apply link.
- **On the hand-run round, scoring is tiered, cheapest first** (`run_round.py` uses no model at all). A mechanical pre-filter, then a small-model knockout
  check against `profile/10-triage-blockers.md`, then full weighted scoring only on survivors, then
  adversarial verification only on postings scoring 70+. Never spend a full weighted evaluation
  discovering a stated requirement the profile cannot meet, such as a licence or a degree. See
  `11-cost-model.md` for the per-stage costs and
  the model to use at each stage.
- **A posting at an employer where a conversation is already running surfaces with a warning,
  never suppressed.** `tools/throttle.py` reads `network_contacts.csv` and attaches
  `warm_route_note` to the shortlist row: "DO NOT APPLY COLD", the person, and what is owed.
  A warm route makes a role *more* worth having, so hiding it to protect the referral would
  lose the role. A standing referral offer sits unused if a cold application to the same
  employer goes in first and spends it for nothing.
- **A reply is what makes a route warm.** `awaiting_reply` means a message went out and
  nothing came back, and a large share of any contact book sits there. Counting them warm would print "DO NOT
  APPLY COLD" over employers where nobody had said a word back, add the referral bonus to
  the fit score and exempt the role from the fit floor, all on the strength of an unanswered
  InMail. `throttle.RESPONDED` is the warm set; `SENT_NO_REPLY` gets
  `outreach_pending_note` instead, which says applying is fine and a second message is not.
  For the outreach note in `documents/applications/`, both halves count: whether they replied
  decides whether *applying* is cold, and decides nothing about sending another message.
- **A referral already in flight is the one contact state that suppresses.** A contact at
  `referral_in_progress` means somebody inside has put the candidate's name forward, so a cold
  application lands as a second, worse record of the same person and the referral is the one
  that loses. `throttle.referral_holds()` blocks every posting at that employer and names who.
  This is what a live process at an employer is recorded as: put the contact on
  `referral_in_progress`, do not rely on remembering it.
- **Company throttle.** A company with an open application in the tracker is suppressed: do not
  surface more roles there. The only override is a role scoring 10+ points higher than what was
  already sent, named explicitly. Where one run turns up several eligible roles at one employer,
  surface only the highest scorer, by its 0-100 fit (priority breaks a tie), and record the rest
  as same-employer duplicates.
- Yield target: at least 5 postings scoring 60+, of which 2 score 75+. A run that misses
  it expands the search and goes again, up to 3 rounds: more title variants, then the
  remaining priority categories, then wider PASS-set locations, then adjacent target sectors
  and named employers, then more pages and the WebSearch fallback.
- If the target is still missed after 3 rounds, say so explicitly with the count, what was
  tried, and the likely blocker. Never pad a thin run with sub-60 postings, and never relax
  a gate or the location filter to hit the number.
- A posting with no stated salary is not penalized. Most postings in most markets omit it.
  Report `not stated`, deduct nothing, and assume the band is met where the employer is
  established with checkable evidence of good pay and benefits. Only a *stated* range whose
  top falls below the walk-away floor gets flagged alongside the score, and even then it
  is a decision input, not an exclusion (the cut line above is the one exception). See "How much
  weight salary actually gets" in `04-job-evaluation.md`.

### Portals and employer boards

- **The app ships two portals, and every other one is the person's.** `linkedin-search`
  and `freehire-search` work in any country and are on by default; a person turns either
  off in `profile/search.py` `PORTALS`, never in its `SKILL.md`, which is the app's.
  LinkedIn's robots.txt disallows its guest job endpoints for crawlers it does not name,
  and `/setup` says so; switching it off is the person's call. A board for one country,
  field or level is built for the person by `/add-portal` (which `/setup` runs for the
  boards its research found), lives in their repo as a personal path
  (`tools/personal_paths.py`), and is searched by every round, kept to the `crawl-delay`
  its `SKILL.md` declares. The app never carries a portal of its own beyond those two
  (`.agents/skills/` is a personal path, and its leak scan fails on one there). What it does
  carry is the market tables (`tools/markets/`, read with `tools/markets.py`): public facts
  about the national, field, graduate, executive, regional and remote boards of many
  countries, each with the robots.txt verdict code made for its search and posting paths
  and the day it was read, which `/setup` reads before it researches anything. A board from
  a table is searched only once `/add-portal` has built it in the person's repo, after
  reading its robots.txt again.
  `python3 tools/portals.py` prints what the next round will search, and the boards
  researched for the person (`PORTAL_RESEARCH`) that have no portal yet.
- **A site that needs a key** (an API such as Adzuna's) shows on the page as what it
  needs ("Needs a free key"; Adzuna "Needs a paid plan") with the steps to get it (`tools/portals.py` `SITES`, `key_help`), and cannot be
  ticked until this computer holds it. When the person says "I have my <site> key", show
  them where it goes the way SETUP.md, "A job site that needs a key", says: never asked
  for in the chat itself, never written into the repo.
- **Employer boards are the third source, and not a portal.** `tools/ats_boards.py` reads
  the employers named in `profile/boards.py` straight from their ATS (every system the
  vendored openroles reads: Greenhouse, Lever, Ashby, Workday, Personio, Teamtailor and the
  rest) or their careers site (`careersite`), which is how large employers that a portal
  under-indexes reach a round,
  and `scrape_wide.py` calls it, so `run_round.py` needs nothing extra. `--probe` re-checks
  every token. A board whose robots.txt disallows it is refused, and so is SmartRecruiters
  by its API's; the reasons are recorded beside each.

## Presenting a round, in full

On the person's page first (CLAUDE.md, "Every output is shown"): the round reaches the page's
Jobs list through `python3 tools/app.py view`. Asked in the chat, or where there is no page,
the chat shows it in full this way too: one line of counts, then the table, then a line only
for what is off. Never more. Asked on the page, the chat says only the counts line and where
it is.

**The round prints the table; never build it from the JSON.** Asked in the chat, run the round
with `--table` (`python3 tools/run_round.py --table`; `--recut --out <the round's
folder> --table` prints it again after a recut). Under `table for the person` it prints the
counts line and the table, columns `# | Score | Verdict | Title | Company | Location | Salary |
Deadline | Check | Apply`, each row's warm route already in Check. Pass both on exactly as
printed, then the lines for what is off (below). Reading the shortlist's JSON to make it costs
about 10k tokens a search and adds nothing.

- **Sorted by score, highest first**, with the score shown. Never alphabetically, by portal or
  by posting date, and never an unscored posting. The printed table already is.
- **The number asked for is filled from the next rows that pass every rule.** The round's cut
  (`run_round.hard_rules`, the one path every row a search brings back goes through) passes
  over a row failing a hard rule (no score, the search window, the candidate's bar with a
  staffing agency, the screen's hard cuts such as a language, a ruling, or a job the person
  already has: applied, drafted, added, kept or still on their list; a place outside theirs, the
  location gate's own fail; an employer judged to clear none of their bar) and takes the next, and says how many it found when it runs out (`shortlist: found 7 of the 10
  asked for`): pass that on. **What it rules out is off the page already**: the round writes each
  as a ruling with its plain reason and puts it on its left-out list, where Bring back undoes it,
  and checks the jobs still on Jobs from earlier searches the same way (`left out: N ...`); the
  page's Found list is read with the same function (`run_round.page_rules`), so a row failing a
  hard rule never shows as a find. Nothing is cleared by hand. A row you find fails a rule the round could not see is ruled out
  with `python3 tools/exclude.py add <url> --title "<title>" --company "<company>" --reason
  "<why>"`, then `python3 tools/run_round.py --out <the round's folder> --recut --show <N>`
  cuts the round again and refills it. Never drop a row from the table by hand: it leaves the
  shortlist short with good rows below it.
- **Every row carries a working apply link**, and so does every posting you later draft
  documents for. The link is the posting URL stored in `seen_jobs.json`, which already
  resolves to the posting itself rather than to a listing page. When documents are produced
  for a posting, repeat its link next to them.
- **Check** says what needs checking before applying, and any red flag: a mass-posting
  pattern ("posted in 6 cities"), a **stated** salary range whose top is below the walk-away
  floor, an unstated on-site expectation, a language level below the posting's bar. An absent
  salary range is not a red flag. A row's warm route from the contact book goes here too
  (`warm_route_note`: "DO NOT APPLY COLD" and who; `outreach_pending_note`): those are the
  referral links. The printed Check holds all of these that the round read; one you find
  in a posting yourself (a mass-posting pattern, an on-site expectation) goes in the line
  under the table, never edited into a row.
- **A line only when something is off**: `skipped (disabled): <portals>` when a portal is
  switched off (`python3 tools/portals.py` says which), so an opt-out is never silent;
  `fallback (websearch): <portals>` when results came from the search fallback, which can be
  stale (omit the line when every portal ran its CLI); one `health: <portal> - <what>` line for each portal the round (or
  `manual-round.md` Step 4.75) found degraded, broken or inconclusive; and when the target was
  missed, a short note: the count against target, the expansions run, the likely blocker and
  the next move. Never pad the table with sub-60 postings.
- **A portal found broken**: after the report, offer to switch it off until it is fixed. For
  the app's two portals (`linkedin-search`, `freehire-search`) that is a `PORTALS` entry in
  `profile/search.py` (`{"freehire-search": False}`), never an edit to their `SKILL.md`,
  which a sync puts back; for a portal built for this person by `/add-portal` it is
  `enabled: false` in that portal's own `SKILL.md` frontmatter. Only with the user's
  confirmation, and never edit anything else in the skill.
- LinkedIn people-search links for a recruiter or a peer at an employer
  (`manual-round.md` Step 4.5) only when the person asks: `/network` is the route to people.

## Important Rules

1. **Never fabricate job postings.** Only present jobs from actual CLI search/detail output or WebSearch/WebFetch results.
2. **Respect deduplication.** Always check seen_jobs.json AND job_search_tracker.csv before presenting.
3. **Focus on configured geographic area.** Skip jobs that require relocation or are clearly outside commute range.
4. **Only open positions.** Skip postings with expired deadlines or those marked as closed. A web-search lead is shown only once its own page proves it is open (`manual-round.md` Step 1c). When this chat's cloud settings block the job sites, no search runs and there is no table: the reply is the fix `python3 tools/env_check.py --say` prints.
5. **Be efficient with detail fetches.** Don't run `detail` or WebFetch on every search hit - pre-filter by title/snippet, then fetch only promising matches.
6. **Parallel searches.** Run portal CLI searches in parallel; use WebSearch only for gaps the CLIs don't cover.
7. **No automated people lookups.** Referral contacts (`manual-round.md` Step 4.5) are LinkedIn search links only - never fetch or scrape LinkedIn people-search result pages programmatically.
8. **Health checks are bounded and honest.** `manual-round.md` Step 4.75 spends at most one probe, one retry, and (in `health` mode) one detail fetch per portal - a diagnosis, not a crawl. A rate-limit is never evidence of breakage. Health verdicts come only from observed CLI output; a portal that could not be tested is reported as inconclusive, never guessed. The on/off switch (`PORTALS` in `profile/search.py`, or `enabled:` in a person's own portal) is the only thing the health check may edit, and only with confirmation.
9. **Score everything you present, and rank by that score.** No posting reaches the user unscored, and the table is ordered by score. A list of unranked links is a failed run.
10. **An under-target run expands before it reports.** `manual-round.md` Step 3.5's expansion ladder runs before the summary is written, up to 3 rounds (`run_round.py --expand` widens the pages itself and prints the other rungs, Step 3.5). Never pad a thin result with sub-60 postings, and never relax a gate or the location filter to make the yield target.
11. **Flag distribution patterns, never accuse.** The mass-posting signal (`manual-round.md` Step 2.5) describes how a listing is being distributed, not a claim that the employer is a scam. Never name a company as fraudulent or untrustworthy - present the observation and let the user decide.
12. **Paginate. Never cap.** A search walks `--page 1..N` until a page comes back empty or wholly duplicated (nothing on it the search's own earlier pages did not have; `scrape_wide.py` PageWalk), and covers every location the PASS set covers, not just the biggest city. `run_round.py` does both; `manual-round.md` Step 1b says how by hand.
13. **Full scoring is one posting per agent, never batched** (Tier 3, `manual-round.md` Step 3), whatever batch size `/rank` uses.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall