具有保护功能的Python BeautifulSoup抓取网页

Question

I am new to Python. My recent project is to get the odd information from a website.

Here is the URL:

http://bet.hkjc.com/default.aspx?url=football/odds/odds_allodds.aspx&lang=EN&tmatchid=120998

I am using Python and BeautifulSoup to process, i cant see any odd data when get the html by

soup.prettify()

The result from above code i got is only logical code, variable and function. i think the page have some protection to the data

what should i do to get the odd information from the protected web page?

Answer 1

It looks like it's not protected, but just generated with javascript. And beautifulsoup can't handle JS. The first work-around most people come to is automating a web browser using something like selenium . You can use this to get the html after the JS has run, and then parse with beautifulsoup as needed.

Answer 2

The answer posted by SuperStew is right but the page loads " http://bet.hkjc.com/football/odds/odds_allodds.aspx?lang=EN&tmatchid=120998 " using JavaScript it is this page that has the data on odds. You didn't state which odds you wanted but the code below is an example of one way to get some data if you want other data you will have to modify it.

import bs4
import requests
url = "http://bet.hkjc.com/football/odds/odds_allodds.aspx?lang=EN&tmatchid=120998"
page = requests.get(url)
soup = bs4.BeautifulSoup(page.text,'lxml')
tOdds = soup.findAll('table', {'class':"tOdds"})
for tOdd in tOdds:
    print (tOdd.text)

Outputs:

  Jong PSV Eindhoven(Home) Draw Jong Utrecht(Away)   1.53 4.00 4.60 
  Jong PSV Eindhoven(Home) Draw Jong Utrecht(Away)   1.97 2.45 4.70 
  Jong PSV Eindhoven[-1](Home) Draw Jong Utrecht[+1](Away)   2.45 3.60 2.26 
  Line High Low  [3/3.5]2.021.70
  Line High Low   [1.5]2.191.60
     1.44    18.00    2.65   
  0 1 2 3 4 5 6 7+   18.00 6.60 4.10 3.65 4.50 6.70 11.00 14.00 
  Odd Even   1.90 1.80 
  H H H D D D A A A   H D A H D A H D A   2.30 14.00 34.00 4.70 6.50 10.50 19.00 14.00 7.50