I'm having difficulty working out how to get Scrapy to observe (and try to achieve) a URL per second limit (e.g. 10 URLs/s)
DOWNLOAD_DELAY seems to be a minimum, so if I set to 0.1 some sites will have a much slower crawl rate than 10 URLs/s. Changing CONCURRENT_REQUESTS doesn't seem to make much difference (e.g. marginal difference between 10 and 100 CONCURRENT_REQUESTS).
My expectation was that a high number of CONCURRENT_REQUESTS would mean getting close to the DOWNLOAD_DELAY. I'm not getting close to memory or CPU limits so I don't think that is the issue.
To try to achieve 10 URLs/s I can set DOWNLOAD_DELAY to a much lower value (or zero) but this means sometimes there are far more r