不要从汤中获取数据

Question

I created bs4 web-scraping app with python. My program return empty list for review. For soup program runs normally.

from bs4 import BeautifulSoup
import requests
import pandas as pd

data = []
usernames = []
titles = []
comments = []

result = requests.get('https://www.kupujemprodajem.com/review.php?action=list')

soup = BeautifulSoup(result.text, 'html.parser')
review = soup.findAll('div', class_="single-review")
print(review)

for i in review:
    header = i.find('div', class_="single-review__header")
    footer = i.find('div', class_="comment-holder")
    username = header.find('a', class_="single-review__username").text
    title = header.find('div', class_="single-review__related-to").text
    comment = footer.find('div', class_="single-review__comment").text
    usernames.append(username)
    titles.append(title)
    comments.append(comment)

data.append(usernames)
data.append(titles)
data.append(comments)

print(data)

It isn't problem with class.

Answer 1

It looks like the reason this doesn't work is because the website needs a login in order to access that page. If in a private tab in a browser you where to visit https://www.kupujemprodajem.com/review.php?action=list , it would just take you to a login page.

There's 2 paths I can think of that you could take here:

1. Reverse engineer how the login process works and use the requests library to make a request to login and get (most likely) the session cookie from that in order to be able to request pages that require sign in.

2. (much simpler) use selenium instead. Selenium is a library that allows you to control a full browser instance, so you would be able to easily input credentials using this method. Beautiful soup on the other hand simply just parses html, so doing things like authenticating often take much more work in Beautiful Soup then they do in Selenium. I'd definitely suggest looking into it if you haven't already.