driver.find_elements_by_xpath('//div[@data-testid="tweet"]') 给出没有 Output

Question

I am trying to scrape tweets using twitter scraper I have tried a lot of solutions but it gives out empty list.

driver.find_elements_by_xpath('//div[@data-testid="tweet"]')

Whole chunk:

cards = driver.find_elements_by_xpath('//div[@data-testid="tweet"]')
if len(cards) > 0:
     card = cards[0]
else:
     raise NoSuchElementException('No cards were found')

error
NoSuchElementException                    Traceback (most recent call last)
/tmp/ipykernel_18368/2095008957.py in <module>
      3      card = cards[0]
      4 else:
----> 5      raise NoSuchElementException('No cards were found')

NoSuchElementException: Message: No cards were found

Screenshot

(It's same as that in twitter scraper github repository: https://github.com/israel-dryer/Twitter-Scraper )

Answer 1

What happens?

You try to select the tweets in element <div> but it is in an <article>

How to fix?

Page need some time to load so wait until first tweet or somthing else is located first and then select more specific with article :

if WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//article[@data-testid="tweet"]'))):
    for e in driver.find_elements(By.XPATH, '//article[@data-testid="tweet"]'):
        print(e.text)