从 quora 中抓取问题答案、日期和投票

Question

I'm trying to scrape answers, dates and upvote figures from this answer using beautifulsoup - however I cannot select the class="pagedlist_item" . The reason I would like to start from this class, which includes the content of each answer, is that some posts don't have upvotes for example so I would end up with lists of elements of different lenghts in case something is missing as well as mixing the order of same variables.

items_soup = BeautifulSoup(html, "html")
items_soup.find_all("div", {"class" : "pagedlist_item"})

when I run this code it returns an empty list - so not sure what's wrong? from this I would then like to extract the text of the answer, the date and the upvote figure (even when there isn't one - so basically replace the empty with a 0).

is it possible to split and get each of the elements I listed? answers text, date of the answer and upvote figure for the answer - the aim is to then create a dataframe.

to keep in mind: the post has 49 answers but doesn't show all of them if you don't scroll down, and I would like to scrape all the 49 answers.

Answer 1

I'm able to get what you're looking for with the following code:

import requests
from bs4 import BeautifulSoup

url = 'https://www.quora.com/What-is-the-brutal-truth-about-data-scientists'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')

question = soup.find('span', {'class': 'ui_qtext_rendered_qtext'})
answers = [ s.text for s in soup.find_all("div", {"class" : "pagedlist_item"}) if s.text ]

results in question == 'What is the brutal truth about data scientists?' and a list of 28 answers.

Answer 2

There is not an empty list when I run the following:

import requests
from bs4 import BeautifulSoup

html ='https://www.quora.com/What-is-the-brutal-truth-about-data-scientists'
r = requests.get(url).text
soup = BeautifulSoup(r, 'html')
soup.find_all("div", {"class" : "pagedlist_item"})

Please check this out! Not sure if you have included requests.