twitter scraper：我无法获取页面上的所有推文，Selenium

Question

I'm trying to build a bot that delete tweets with a specific date, so I had to scroll the page to fetch more tweets every time, that's not the problem, the problem is when I'm trying to get the tweets info, it's only fetch the last five tweets, when I try to scroll the page by this line of code

driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")

it leads to a gap between the tweets date

output example{not a real one}: 1- tweet1 date = 22 jan 23.... 5- tweet5 date = 20 jan 23 6- tweet6 date = 13 dec 22

there is a gap between output 5 and 6,

How Could I load all the tweets on the page, before scrolling it?

# Go to profile page

driver.get("``https://twitter.com/MYACC``")

# load tweets

tweets = driver.find_elements(BY.XPATH, "//article[@data-testid='tweet']")

last_height = driver.execute_script("return document.body.scrollHeight")

while True:

    for tweet in tweets:
        
        print(tweet.text)
         driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")
    
    new_height = driver.execute_script("return document.body.scrollHeight")
    
    time.sleep(5)
    
    tweets = driver.find_elements(BY.XPATH, "//article[@data-testid='tweet']")
    
    if last_height == new_height:
        break

driver.quit()

this is the code

I tried to scroll the page by 5000 pixel, but the problem is that, its stops on cretin point, and start running from the start..

Answer 1

As we will see in a moment there is no way to load all the tweets on the page. The following GIF shows the HTML code of the elements (tweets and other things) contained in the timeline as I scoll down the page. Each element can be identified by its transform style attribute. The first tweet has translateY(0px) , the second tweet has translateY(735.6px) and so on. As I scroll down you can see that the first tweets disappear from HTML and new ones appear at the bottom.

From the point of view of scraping, scrolling can be messy. A better and cleaner strategy is to delete each tweet from the HTML after you are done with it. New tweets will automatically get loaded as if you were scrolling.

driver.get(...)

# wait for tweets to appear in the page
first_tweet = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[data-testid=cellInnerDiv]")))
# scroll to first tweet
driver.execute_script('arguments[0].scrollIntoView({block: "center", behavior: "smooth"});', first_tweet)
time.sleep(1)

date_for_deletion = '2022-05-21'
stop_condition = 50

for i in range(999):
    tweets = driver.find_elements(By.CSS_SELECTOR, 'div[data-testid=cellInnerDiv]')
    is_a_tweet = tweets[0].find_elements(By.XPATH, './/article[@data-testid="tweet"]')
    
    # check if list is not empty (i.e. check if element is a tweet or a "who to follow" element)
    if is_a_tweet:
        date = tweets[0].find_element(By.XPATH, './/time').get_attribute('datetime').split('T')[0]
        print(date, is_a_tweet[0].text[:90].replace('\n',''))
        if date == date_for_deletion:
            # delete the tweet
            ...
        time.sleep(1)

    # delete element from HTML
    driver.execute_script('var element = arguments[0]; element.remove();', tweets[0])
    if i == stop_condition:
        break

You will get an output such as

2022-05-30 Text of tweet 1
2022-05-28 Text of tweet 2
2022-05-27 Text of tweet 3
...