AttributeError: 'NoneType' object has no attribute 'get' helpp

Question

I need to take link to the latest user tiktok video. it worked, but then stopped working

import bs4
import requests
from fake_useragent import UserAgent
from bs4 import BeautifulSoup

link = "https://www.tiktok.com/@dava_m?"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36 OPR/81.0.4196.61'}
full_usd_page = requests.get(link, headers=headers).text
soup = BeautifulSoup(full_usd_page, 'html.parser')
product_link = soup.find("a", class_="").get("href")
print(link)

product_link = soup.find("a", class_="").get("href")
AttributeError: 'NoneType' object has no attribute 'get'

it returns none, but an hour ago it worked. pls help to get latest video link https://www.tiktok.com/@dava_m ?

Answer 1

This means soup.find("a", class_="") is returning a nonetype object. Try running it by itself without the get bit and check the output. Make sure the output of the following is not None.

soup.find("a", class_="")

It's possible that the function can't find anything.

Answer 2

Probably the page is dynamically generated, and it's not sufficient to use simply requests . But other methods are quite more sophisticated. They might be selenium , scrapy-splash , etc. Search the web for python scrape dynamic content/page .