Downloading csv file from website with buttons?
Hello,
I am trying to download the following table in a csv format: http://stats.oecd.org/Index.aspx?DataSetCode=MEI_CLI
However this website does not provide a direct link. Rather you have to press a few buttons before you are allowed to download the file.
Is there a way to download this file automatically from within R?
Thanks a lot for any suggestions in advance!
Posted Apr 9, 2017, 10:07 PM UTC
u/SataMaxx has the right idea, but I don't think it works in this case. The final GET call:
http://stats.oecd.org/Download.ashx?type=csv&Delimiter=%2c&IncludeTimeSeriesIdentifiers=False&LabelType=CodeAndLabel&LanguageCode=en
doesn't work because its downstream to the POST call:
http://stats.oecd.org/modalexports.aspx?exporttype=bulk&FirstDataPointIndexPerPage=undefined&SubSessionId=801bd869-896a-499d-a3e5-4c9b12d14783&Random=0.12979474325124074
Notice the subSessionId and Random parameters. You're not going to be able to guess those, and you don't even want to look at what's in the body of this POST call.
Trying to hunt down the right xpath/selector from the original url:
http://stats.oecd.org/Index.aspx?DataSetCode=MEI_CLI
won't work either because your data is not static html. It's dynamically triggered by javascript, so you need to actually be browsing for the data to render. For what its worth, the right minimal selector would be: "#fixedheader > table"
There IS an option here: headless browsing. This will allow you to traverse the website and click on the buttons programatically. From experience this is a gigantic pain in the ass, but it can be done with PhantomJS or even natively in R with RSelenium (an R package that provides a Selenium API).
Thanks a lot for all the info. Looks like I am in trouble then but I suspected that much.
Sounds like you are knowledgable in this area. I was thinking of an alternative way to do it by extracting the table directly from the html (it is displayed before downloading the file). Would that be doable in your opinion?
The problem is that table isn't actually in the html to begin with. The html is created after you start browsing it via javascript. Notice how the data loads after all the other elements on the page - thats because its not really part of the original html. If you disable javascript on your browser you can see that the data never appears on your browser.
I supplied the right selector above if you want to give it a shot, but because the js isn't triggered when you do standard webscrapping, the html won't be there either.
I have a potential solution I'm looking into that won't require headless browsing but it looks super ugly at the moment. I'll update when I have more.
Edit: Well, that ugly solution is XML, and there's no way in hell I am putting myself through the trauma of parsing XML, so headless browsing it is!
Thank you again for sharing your expertise and providing some insight into this issue. As mentioned in another comment - it's late here so I am going to sleep. But will try your proposed ideas in the morning.