Real-time web content for RAG/chat pipelines in 2026?

How are you all scraping sites at scale?

My Brave API + Crawl4AI setup is blocked by at least 80% of sites. Falling back to Brave snippets are too thin for good answers.

Does Cloudflare /crawl solve this?

What's working these days?