使用beautifulsoup4进行抓取时数据丢失

Question

Actually I'm a newbie to the parsing stuff with Python Beautifulsoup4. I was scraping this website . I need Current Price Per Mil on the front page.

I already spent 3 hours with this. While looking for the solution on internet. I got to know that there is a library PyQT4 that can mimic like a web browser and load the content and then once it's done with loading you can extract ur required data. But I got crashed.

Used this approach to collect the data in raw text format. I tried other approaches too.

def parseMe(url):
    soup = getContent(url)
    source_code = requests.get(url)
    plaint_text = source_code.text
    soup = BeautifulSoup(plaint_text, 'html.parser')
    osrs_text = soup.find('div', class_='col-md-12 text-center')
    print(osrs_text.encode('utf-8'))

Please have a look on this image . I think the problem is with ::before and ::after tags. They appear once the page get loaded.
Any help will be highly appreciated.

Answer 1

You should use selenium instead of `requests:

from selenium import webdriver
from bs4 import BeautifulSoup

def parse(url):
    driver = webdriver.Chrome('D:\Programming\utilities\chromedriver.exe')
    driver.get('https://boglagold.com/buy-runescape-gold/')
    soup = BeautifulSoup(driver.page_source)
    return soup.find('h4', {'id': 'curr-price-per-mil-text'}).text

parse()

Output:

'Current Price Per Mil: 0.80USD'

The reason is that the value of that element is obtained through JavaScript, which requests can't handle. This particular snippet of code uses the Chrome driver; if you prefer, you can use the Firefox/some other browser equivalent (you will need to install the selenium library and look for the Chrome driver yourself).

Answer 2

The web page makes an XHR to fetch a JSON file with the but price in it

import requests

r = requests.get('https://api.boglagold.com/api/product/?id=osrs-gold&couponCode=null')
j = r.json()
# print(j)
print('sellPrice', j['sellPrice'])
print('buyPrice', j['buyPrice'])

Outputs:

sellPrice 0.8
buyPrice 0.62

Answer 3

As mentioned by the other answers, this page only contains the text Current Price Per Mil: and 0USD . The value in the middle - 0.8 - is obtained dynamically with JS from the url described below (which can be obtained using a process described (for example) here and many other places . That site checks for bots so you have to use a method described (for example) here .

So all together:

url = 'https://api.boglagold.com/api/product/?id=osrs-gold&couponCode=null'
import requests
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'})

response.json()['sellPrice']

Output:

0.8

Answer 4

The issue is that the javascript dynamically adds the data you want to scrap on that website. You could try to run JS on the client side, wait for fetching the data you want to scrap and then get the DOM contents - if you want to do it that way, please look at @gmds answer to this question. The other method is to check what requests the javascript code is making and which one contains the information you need. Then you can make that request(s) using python and get the required data without the need of using PyQT4 or even BS4.