使用 BeautifulSoup 和 RegEx 或 Lambda 循环访问 URL 以进行匹配？

Question

I am trying to loop through a few URLs and scrape out one specific class. I believe it's called:

<div class="Fw(b) Fl(end)--m Fz(s) C($primaryColor" data-reactid="192">Overvalued</div>

Here is the URL:

https://finance.yahoo.com/quote/goog

Here is the data that I want for GOOG.

Near Fair Value

I believe this will require some kind of Lambda function or RegEx. I tried to do this without using these methodologies, but I couldn't get it working. Here is the code that I am testing.

import requests
from bs4 import BeautifulSoup
import re

mylink = "https://finance.yahoo.com/quote/"
mylist = ['SBUX', 'MSFT', 'GOOG']
mystocks = []

html = requests.get(mylink).text
soup = BeautifulSoup(html, "lxml")

#details = soup.findAll("div", {"class" : lambda L: L and L.startswith('Fw(b) Fl(end)--m')})

details = soup.findAll('div', {'class' : re.compile('Fw(b)*')})
for item in mylist:
    for r in details:
        mystocks.append(item + ' - ' + details)

print(mystocks)

Here is a screen shot:

After the code runs, I would like to see something like this.

GOOG - Near Fair Value
SBUX - Near Fair Value
MSFT - Overvalued

The problem is, that if I use something like this: 'Fw(b)*' , I get too much data pulled back. If I try to expand that, to this: 'Fw(b) Fl(end)--m Fz(s)' , I get nothing back. How can I get the results I showed above?

Answer 1

No need to use regex, CSS selector is enough. The key is to use correct HTTP header - User-Agent .

For example:

import requests
from bs4 import BeautifulSoup

urls = [('GOOG', 'https://finance.yahoo.com/quote/goog'),
        ('SBUX', 'https://finance.yahoo.com/quote/sbux'),
        ('MSFT', 'https://finance.yahoo.com/quote/msft')]

headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

for q, url in urls:
    soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')
    value = soup.select_one('div:contains("XX.XX") + div').text

    print('{:<10}{}'.format(q, value))

Prints:

GOOG      Near Fair Value
SBUX      Near Fair Value
MSFT      Overvalued

Answer 2

The issue is that the html being returned from your requests get-request and from a request in Broswer, eg view-source:https://finance.yahoo.com/quote/goog are different. The div that has your target class is missing when you visit the page using python. I found that out by printing the html from requests and comparing with the browser html.

Here are the suggested steps to take:

1. Append the link ending to each url by looping through mylist

2. The Yahoo server detects that you are a robot by reading your request's headers and limits some information. You need to add relevant headers to disguise your request.

3. I suspect that the yahoo server is only reading your user-agent but I'll leave that for you to experiment; and its good for me to post the full headers here for reference purposes.

The headers can be got from the chrome dev-tools, in the network tab. Trillworks Online tool shows how to do so and helps you convert them to requests code.

Proposed solution:

import requests
from bs4 import BeautifulSoup

mylink = "https://finance.yahoo.com/quote/"
mylist = ['SBUX', 'MSFT', 'GOOG']
mystocks = []

headers = {
    'authority': 'finance.yahoo.com',
    'cache-control': 'max-age=0',
    'dnt': '1',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'sec-fetch-site': 'none',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-user': '?1',
    'sec-fetch-dest': 'document',
    'accept-language': 'en,en-US;q=0.9,fr-FR;q=0.8,fr;q=0.7,ar-EG;q=0.6,ar;q=0.5,my-ZG;q=0.4,my;q=0.3',

    'cookie': '', # Note: I removed the cookie value, it was too long
}

for item in mylist:
    html = requests.get(mylink + item, headers=headers).text
    soup = BeautifulSoup(html, "lxml")
    details = soup.find('div', class_="Fw(b) Fl(end)--m Fz(s) C($primaryColor")
    mystocks.append(item + ' - ' + details.text)

print(mystocks)

This prints:

GOOG - Near Fair Value
SBUX - Near Fair Value
MSFT - Overvalued