如何使用Python 3登录网站并进行抓取

Question

I would like to log-in to facebook messenger and parse the HTML.

import requests
from bs4 import BeautifulSoup
import webbrowser
page = requests.get("https://www.messenger.com", auth=
('username', 'password'))

soup = BeautifulSoup(page, 'html.parser')

print(soup)

I got this from another stack question but it is throwing me this error:

    File "C:/Code/Beautiful Soup Web Scraping.py", line 7, in <module>
    soup = len(BeautifulSoup(page, 'html.parser'))
  File "C:\Users\Ethan\AppData\Local\Programs\Python\Python37\lib\site-packages\bs4\__init__.py", line 246, in __init__
    elif len(markup) <= 256 and (
TypeError: object of type 'Response' has no len()

How can I get this to work?

Answer 1

You must pass to BeautifulSoup the content of the web page, not the Response object returned by requests.get . To get the content use the Response.content property.

In your example use : soup = BeautifulSoup(page.content, 'html.parser')

Answer 2

I would recommend using Selenium , which will allow you to login to Facebook, navigate to the desired page, and retrieve the html. You can then pass the HTML to BeautifulSoup . Take a look at this blog post to get started.