从Web下载Python下载csv文件

Question

Goal : want to automatize the download of various .csv files from http://www.tocom.or.jp/historical/download.html using Python (this is not the main issue though)

Specifics : in particular, I am trying to download the csv files for the "Tick Data" (fifth heading from the bottom, for the 5 days available.

Problem : when I see the source code for this web page, look for "Tick Data" I see the references to these 5 .csv files but they're not with the usual href tag. As I am using Python (urllib) I need to know the URLs of these 5 .csv files but don't know how to get them.

This is not a question of Python per se, but about how to find the URL of some .csv that can be downloaded from a web page. Hence, no code is provided.

Answer 1

The page uses JavaScript to create the URL:

<select name="tick">
  <option value="TOCOMprice_20121122.csv">Nov 22, 2012</option>
  <option value="TOCOMprice_20121121.csv">Nov 21, 2012</option>
  <option value="TOCOMprice_20121120.csv">Nov 20, 2012</option>
  <option value="TOCOMprice_20121119.csv">Nov 19, 2012</option>
  <option value="TOCOMprice_20121116.csv">Nov 16, 2012</option>
</select>
  <input type="button" onClick="location.href='/data/tick/' + document.form.tick.value;" 
        value="Download" style="width:7em;" />

It combines a path, that the browser will use against the current site. So each URL is:

http://www.tocom.or.jp + /data/tick/ + TOCOMprice_*yearmonthday*.csv

By the looks of it, the data only covers weekdays.

These are easy enough to cobble together into automated URLs:

import requests
from datetime import datetime, timedelta

start = datetime.now() - timedelta(days=1)
base = 'http://www.tocom.or.jp/data/tick/TOCOMprice_'

next = start
for i in range(5):
    r = requests.get(base + next.strftime('%Y%m%d') + '.csv')
    # Save r.content somewhere
    next += timedelta(days=1)
    while next.weekday() >= 5:  # Sat = 5, Sun = 6
        next += timedelta(days=1)

I used requests for it's easier-to-use API, but you can use urllib2 for this task too if you so wish.

Answer 2

Use Chrome w/Dev Tools, Firefox w/Firebug or Fiddler to look at the request URL when you hit the download button.

(for example, I see this for Nov 22: http://www.tocom.or.jp/data/tick/TOCOMprice_20121122.csv )

Answer 3

You can determine the download link Using the developer menu of your browser, among other things. I use chrome, and I'm shown that the link is

http://www.tocom.or.jp/data/souba_d/souba_d_20121126_20121123_0425.csv

That URL structure seems pretty straight forward to guess, and another link right on the page:

http://www.tocom.or.jp/historical/keishiki_souba_d.html

Indicates how to structure pulls. A good bet is just to structure the csv pulls in 5 minute intervals.

Good luck!