如何从 Twitter 打印推文？

Question

I am trying to scrape tweets from twitter for a side project.

Having difficulty with outputs.

Using latest version of pycharm.

import urllib
import urllib.request
from bs4 import BeautifulSoup

theurl = "https://twitter.com/search?q=ghana%20and%20jollof&src=typed_query"
thepage = urllib.request.urlopen(theurl)


soup = BeautifulSoup(thepage, "html.parser")
i = 1
for tweets in soup.findAll('div', {
    "class": "css-901oao css-16my406 r-1qd0xha r-ad9z0x r-bcqeeo r-qvutc0"
}):
    print (i)
    print (tweets.find('span').text)
    i = i+1
    print(tweets)

I do not receive any errors at all but there no outputs for the tweets.

Answer 1

You should use the requests library, and also you are missing user-agent header in your request which seems to be mandatory for twitter.

Here is a working example:

import requests
from bs4 import BeautifulSoup

# without this you get strange reponses
headers = {
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
}

# the correct way to pass the arguments
params = (
    ('q', 'ghana and jollof'),
    ('src', 'typed_query'),
)

r = requests.get('https://twitter.com/search', headers=headers, params=params)
soup = BeautifulSoup(r.content, 'html.parser')
allTweetsContainers = soup.findAll("div", {"class": "tweet"})

print(len(allTweetsContainers))
# all that remains is to parse the tweets one by one

Problem is that this way you will load only 20 tweets per request you will need to examine the network tab and see how the browser loads the rest dynamically.

This however is very tedious, I strongly recommend using a library that directly calls the twitter api, like https://github.com/twintproject/twint