A modern way to mirror websites?
I happen to archive websites from time to time. Years ago, on Windows, I used Offline Explorer for that. It was a pretty decent tool, which usually downloaded all resources. Probably it does not do that well anymore.
On Linux we have wget -r and httrack, both with simple home brew HTML parsers. With all my sympathy for wget, they are just useless in these days of dynamic script loading and writing JS-heavy websites. A real web browser engine is needed to successfully mirror a site.
So my question is: which Linux programs or browser extensions are able to mirror a site built by 2013 standards?
EDIT: This seems to confuse people. I'm looking to save all interlinked pages in a domain, not to press ctrl-s in my browser.
Posted Sep 18, 2013, 1:19 PM UTC · edited
You're asking for scraping, not mirroring.
As far as I understand, scraping = extracting data from parsed pages. Like going through a store and extracting names and prices to a database.
I want mirroring, a.k.a. spidering, which just saves all subpages of a website, along with resources needed to render them.
To me mirroring sounds more like proxying+caching somehow (which is why Squid was mentioned). You want to crawl+scrape the site.
I'm working on a crawler/scraper that should copy all pages/css/js/images to a directory (or rather a git repository). It's far from actually stable and full of hacks, but I'm using Zombie.js at the moment: http://zombie.labnotes.org/ Zombie.js is a programmable browser mostly used for testing websites. It is built on top of JSDom. JSDom is a implementation of a real browser DOM. That means it can (as far as it's implemented) load pages, run its javascript and parse CSS. JSDom is built on top of node.js.
You could also check out PhantomJS, which is a headless+scriptable webkit browser. I'd think there are scripts that should be able to download all files and follow all links in PhantomJS.
If you build that out to call all assets on a page and execute the javascript and you run this as an actual crawler and not just as a one-off thing you're going to cause a lot of headaches for people running larger sites and get yourself blocked from sites who pay attention to things like ad metrics or analytics from javascript calls.
If you insist on doing it please for the love of god set your useragent to something that shows it's a crawler and not try to pretend you're a real browser.
Yes, other people are already doing it, and more people will I'm sure. Just venting because it causes lots of issues when someone suddenly starts crawling hundreds of thousands of webpages and sends javascript events that they clicked all the ads on all the pages.
Indeed, there is no intention to anger/break sites. It uses the useragent of ZombieJS atm. It doesn't yet have delays in crawling, but I want to add that too. The crawling doesn't have to be as fast as possible and it's used mostly for smallish static sites. A limit on the number of pages/assets/MB/etc will also come.
[deleted]