Python网站搜寻器，使用Scrapy保存文件

Question

I'm attempting to write a crawler that will take a certain search entry and saving a whole bunch of .CSV files correlated to the results.

I already have the spider logging in, parsing all the html data I need, and now all I have left to do is figure out how I can save the files I need.

So the search returns links such as this https://www.thissite.com/data/file_download.jsp?filetype=1&id=22944

Which then in a web browser prompts you to save the correlated .csv file. How can I write my spider to be able to load this page and download the file? Or is there a way I can catch a static link to the information?

Answer 1

If you crawled the link to the CSV files you can simply download them with wget, which is able to login to a page too.

You either specify the --http-user and --http-passwd, or you use cookies as follows:

$ wget --cookies=on --keep-session-cookies --save-cookies=cookie.txt --post-data "login=USERNAME&password=PASSWORD" http://first_page
$ wget --referer=http://first_page --cookies=on --load-cookies=cookie.txt --keep-session-cookies --save-cookies=cookie.txt http://second_page

It depens on how your site handles logins. There are a few other ways to login to a page with wget, I'm sure you find those by googling.

I'd suggest doing all this in a special Scrapy Pipeline , so it's all done in Scrapy and not in an external script.