Beautifulsoup for loop不能获取所有元素

Question

I want to scrape tweets from some twitter posts, I use for that BeautifulSoop library.what I want to do is to get the original post and all the replies if there is any replies (but all of them). I managed to get the original post, and I wrote this loop to get me all the replies, but it returns me only the first one. Any help please thanks !

from bs4 import BeautifulSoup
import urllib.request

url= "https://twitter.com/20Minutes/status/692778440211169280"

list_Original_message =[]

readfile=urllib.request.urlopen(url).read()
soup = BeautifulSoup(readfile)

# ..... the first part of my script is set to scrape the original post, I omit it # because it works!

# loop to get the replies :

replies = soup.find_all('ol',{"class":'stream-items js-navigable-stream'})
for m in replies :
    name = m.findAll('strong',class_="fullname js-action-profile-name show-popup-with-id")[0]
    print(name.string)
    profile = m.findAll('span',class_="username js-action-profile-name")[0]
    print(profile.get_text())
    link = m.findAll('a',class_="tweet-timestamp js-permalink js-nav js-tooltip")[0]['href']
    print('https://twitter.com'+link)
    time = m.findAll('a',class_="tweet-timestamp js-permalink js-nav js-tooltip")[0]['title']
    print(time)
    message = m.findAll('p',class_="TweetTextSize js-tweet-text tweet-text")[0]
    print (message.get_text())

This is the result I get, only the first reply :

Mais l'eau dit

@Queen_MeloMau

https://twitter.com/Queen_MeloMau/status/692797851139710978

11:54 AM - 28 Jan 2016

@20Minutes dites moi que c'est une blagounette la @slavicdelrey

Answer 1

Only the first (few) tweets are actually sent to you to your original request, the rest gets loaded asynchronously. Use the Twitter APIs, they're there for a reason.