美丽的汤 .find 找不到任何东西

Question

I am trying to scrape posts in a Facebook group:

URL = 'https://www.facebook.com/groups/110354088989367/'

headers = {
    "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'
}


def checkSubletGroup():
    page = requests.get(URL, headers=headers)
    soup = BeautifulSoup(page.content, 'html.parser')
    posts = soup.find_all("div", {"class_": "text_exposed_root"})
    print(soup.prettify())
    for post in posts:
        print(post)


checkSubletGroup()

The div with class="text_exposed_root" is clearly there because I can find it with CTRL f when I search in print(soup.prettify()) , but when I do soup.find_all("div", {"class_": "text_exposed_root"}) it is returning an empty list, so are many other class names that are clearly there.

Please help.

Answer 1

The problem is that all those <div> are inside a commented out HTML block.

Something like this can workaround the issue:

soup = BeautifulSoup(page.text.replace('<!--', '').replace('-->', ''), 'html.parser')

After that you can simply do:

posts = soup.find_all('div', 'text_exposed_root')

I hope it helps.

Answer 2

You only need to use class_ when you're checking the class using a keyword argument, because class is a Python reserved word and can't be used as a variable. If you pass the attributes as a dictionary, you just use class .

So it should be

posts = soup.find_all("div", {"class": "text_exposed_root"})

or

posts = soup.find_all("div", class_ = "text_exposed_root")