Scrape website data for CSV

Question

Rather inexperienced with this type of programming effort, much more familiar with embedded systems. I have very little web programming xp.

What I'd like to achieve:

A website (danglefactory.com) has a great table of statistics that I'd like to download into a CSV for processing. On the website, there is a button that calls an internal script to craft a cvs and prepare for download.

Referer http://www.danglefactory.com/projections/skaters/daily

Script http://www.danglefactory.com/scripts/copy_csv_xls.swf

I prefer a python solution, that will be able to fetch this csv either to temp or local storage for processing.

Thanks in adv.

Answer 1

First approach you can take is pretty low-level.

Under the hood, there are JSON API calls that you can simulate using, for example, requests .

Here is how you can get the daily projections:

import requests

url = 'http://www.danglefactory.com/api/DailySkaterProjections?_=1415200157912'
response = requests.get(url)

data = response.json()
print data

Prints:

[{u'A': 0.61,
  u'Blocks': 0.37,
  u'Corsi': 0.53,
  u'FOL': 9.07,
  u'FOW': 8.95,
  u'FOWinPerc': 49.6,
  u'G': 0.39,
  u'Giveaways': 0.89,
  u'Hits': 0.54,
  u'Name': u'John Tavares',
  u'Opponent': u'ANA',
  u'P': 0.99,
  u'PIM': 0.51,
  u'PPA': 0.24,
  u'PPG': 0.11,
  u'PlayerID': 411,
  u'PlusMinus': 0.05,
  u'PrimaryPosition': u'C',
  u'SHA': 0.0,
  u'SHG': 0.0,
  u'ShPerc': 12.6,
  u'Shots': 3.1,
  u'TOI': 20.39,
  u'Takeaways': 0.82,
  u'Team': u'NYI'},
 {u'A': 0.7,
  u'Blocks': 1.0,
  u'Corsi': 0.47,
  u'FOL': 8.69,
  u'FOW': 8.43,
  u'FOWinPerc': 49.3,
  u'G': 0.28,
  u'Giveaways': 0.84,
  u'Hits': 1.49,
  u'Name': u'Ryan Getzlaf',
  u'Opponent': u'NYI',
  u'P': 0.97,
  u'PIM': 0.68,
  u'PPA': 0.22,
  u'PPG': 0.07,
  u'PlayerID': 161,
  u'PlusMinus': 0.06,
  u'PrimaryPosition': u'C',
  u'SHA': 0.04,
  u'SHG': 0.02,
  u'ShPerc': 11.9,
  u'Shots': 2.3,
  u'TOI': 20.52,
  u'Takeaways': 0.61,
  u'Team': u'ANA'},

  ...

}]

Then, you can convert the results into csv accordingly using csv module.

---

Another solution could be to use selenium browser automation tool, but the problem is that the CSV button and the table is inside a Flash object which selenium cannot interact with .

---

You can though use an image recognition and screen automation tool like sikuli to find that CSV button and click on it. This is if you still want to stay on the "high-level".

Hope that helps.