如果网站上的元素发生变化，为什么我的 Beautifulsoup 元素保持不变？

Question

import requests, bs4
import time


def get_tesla_price():
    website = "https://finance.yahoo.com/quote/TSLA/"
    res = requests.get(website)
    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    elem = soup.select("span[class='Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)']")

    price = elem[0].getText()
    print(price)

for _ in range(10):
    get_tesla_price()
    time.sleep(2)

Above is some code which should scrape the current value of the tesla stock. If you run it you will see that price stays the same for all 10 function calls. But if you check the website in website you see that the price of the stock changes almost every second.

Why, every time when i call my function, price is the same value?

Answer 1

This is because of your user agent, yahoo detects you are a robot, add user agent in headers:

def get_tesla_price():
    website = "https://finance.yahoo.com/quote/TSLA/"
    res = requests.get(website, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'})
    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    elem = soup.select("span[class='Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)']")

    price = elem[0].getText()
    res.close()
    print(price)

Or you can do it with just one module requests-html:

from requests_html import HTMLSession
import time

session = HTMLSession()

def get_tesla_price():
    website = "https://finance.yahoo.com/quote/TSLA/"
    r = session.get(website)
    price = r.html.find("span[class='Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)']", first=True)
    print(price.text)
    

for _ in range(20):
    get_tesla_price()
    time.sleep(1)

output:

682.66
682.66
682.53
682.53
682.53
682.65
682.62
682.69
682.65
682.71
682.71
682.83
682.83
682.83
682.83
682.79
682.79
682.79
682.79
682.80