**Lupin - An adaptative web scraper with smart escalation**

[Github repo link](https://github.com/vnguyen42/lupin)

Here’s a project I built for personal usage and thought could help others people.  
  
Most web pages don't need a stealth browser. But when they do, you shouldn't have to figure that out yourself. 

A recurring problem I’ve run into a lot of my scraping projects is that HTTP requests works 80% of the time, but fails on protected websites (Cloudflare / DataDome), and sometimes, a regular scraper I've set up with http works for a few days then suddently start getting blocked.

The usual workaround is to move the whole scraping pipeline to a stealth browser, but that means having to pay the full browser cost to cover the 15-20% of failing cases, meaning: more resources usage (RAM, CPU), a lot slower responses on average an and much higher proxy costs (can easily be 5-10 more bandwidth per page loaded for a full rendering).

Lupin is a scraper that takes a different hybrid opionated approach: it starts with the cheapest approach first (HTTP) then escalates to a stealth browser(s) when the page is blocked (empty content detected, cloudflare/datadome wall or 403). It also remembers which extraction method (http/browser) worked per domain for a rolling period (24 hours by default).

At the moment, Camoufox and Patchright have the highest pass rate, so the escalation looks like this:

HTTP (fast, \~0.2s) ──→ Blocked ? ──→ Camoufox (stealth Firefox) ──→ Blocked ? ──→ Patchright (stealth Chrome)

In a practical use, this allows to bypass most anti-bot protections on medium-to-hard websites, and you can plug in a proxy if needed to handle the rest at higher volume.

In my personal usage, I was able to plug it inside a news scraper bot and cover 90% of the popular press I was targeting, with good volume.

The features of the scraper are:

* Auto-escalation
* Clean Markdown
* Structured JSON extraction (w/ LLM)
* Built-in social media scrapers
* Crawl
* Proxies
* Can be used as a CLI, Node library, as MCP or Docker

Any feedback are welcome.

I built a scraper that escalates from HTTP to browsers