Structured data out of pages that would rather you didn't.
Extraction that survives JS rendering, rate limits, and layout changes, and validates what it pulled, so you get clean records instead of silent garbage.
Drive it — point it at a cooperative target, then a hostile one, and watch it survive the block and refuse to ship the rows that don't validate.
The scraping is easy. The fighting back is the job.
What you just operated is one pass through this. A cooperative page walks straight down the spine. A hostile one rate-limits the fetch and drifts its markup, so the interesting engineering is the backoff loop and the validation gate, not the CSS selector.
Polite pacing, and on a 429 it backs off and rotates through a proxy pool, not gives up.
A real headless browser when the data only exists after the page's own scripts run.
Resilient selectors pull each field out of the markup into raw values.
Every row is checked against a schema. This is the gate that stops garbage.
Matches ship as typed records; drift is parked with the reason, never shipped.
A row ships only if it matches.
The difference between a scraper you can trust and one you can't is this gate. Extracted values are coerced to real types and checked against a schema. A row that's missing a price or has the wrong shape doesn't get a null and ship anyway — it's quarantined with the reason.
1// listing.schema — a row ships only if it matches2{3 id: string(), // kept as the key4 name: string().min(1),5 price: number().positive(), // "$1,395" -> 13956 currency: enum(["USD", "EUR"]),7 in_stock: boolean(), // "In stock" -> true8}Most of a good scraper is the error handling.
Anyone can select a CSS node on a good day. The value is in the run that survives a block, notices the layout moved, and tells you instead of quietly returning junk.
A real browser when the data only exists after the page's scripts run.
Backoff, proxy rotation, and human-like pacing, so a 429 is a pause, not a failure.
Every row checked against a schema. Drift is caught and quarantined, never shipped silently.
Follows every page and collapses duplicates, so you get the whole set, once.
When the site's layout shifts enough to break extraction, you hear about it, fast.
Only fetches what changed since last time, so a daily pull stays cheap and polite.
A scraper that ships a null where the price should be is worse than no scraper.
Bad data is expensive precisely because it looks fine until something downstream trusts it. So the whole design points one way: survive the block, catch the drift, and hold back anything that doesn't validate rather than quietly poisoning the dataset. We also build scrapers that respect robots.txt and a site's terms where they apply, and we'll tell you when a target is off-limits rather than quietly working around it.
Clean, typed records, even from a page that resists.
Survives JS rendering, rate limits, and layout drift, validates every row against a schema, and quarantines what doesn't match instead of shipping garbage.