抓取雅虎股票新闻

Question

I am scraping news articles related to Infosys at the end of page but getting error selenium.common.exceptions.InvalidSelectorException: Message: invalid selector. Want to scrape all articles related to Infosys.

from bs4 import BeautifulSoup
import re
from selenium import webdriver
import chromedriver_binary
import string
import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webdriver.Chrome("/Users/abhishekgupta/Downloads/chromedriver")
driver.get("https://finance.yahoo.com/quote/INFY/news?p=INFY")

for i in range(20): # adjust integer value for need
       # you can change right side number for scroll convenience or destination 
       driver.execute_script("window.scrollBy(0, 250)")
       # you can change time integer to float or remove
       time.sleep(1)

print(driver.find_element_by_xpath('//*[@id="latestQuoteNewsStream-0-Stream"]/ul/li[9]/div/div/div[2]/h3/a/text()').text())

Answer 1

You could use less detailed xpath using // instead of /div/div/div[2]

And if you want last item then get all li as list and later use [-1] to get last element on list

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/abhishekgupta/Downloads/chromedriver")
#driver = webdriver.Firefox()

driver.get("https://finance.yahoo.com/quote/INFY/news?p=INFY")

for i in range(20):
       driver.execute_script("window.scrollBy(0, 250)")
       time.sleep(1)

all_items = driver.find_elements_by_xpath('//*[@id="latestQuoteNewsStream-0-Stream"]/ul/li')

#for item in all_items:
#    print(item.find_element_by_xpath('.//h3/a').text)
#    print(item.find_element_by_xpath('.//p').text)
#    print('---')
    
print(all_items[-1].find_element_by_xpath('.//h3/a').text)
print(all_items[-1].find_element_by_xpath('.//p').text)

Answer 2

xPath you provided does not exist in the page.

Download the xPath Finder Chrome Extension to find the correct xPath for articles.

Here is an example xPath of articles list, you need to loop through id:

/html/body/div[1]/div/div/div[1]/div/div[3]/div[1]/div/div[5]/div/div/div/ul/li[ID]/div/div/div[2]/h3/a/u

Answer 3

I think your code is fine just one thing: there are few difference when we retrieve text or links when using xpath in selenium as compare to scrapy or if you are using lxml fromstring library so here is something that should work for you

#use this code for printing instead 
print(driver.find_element_by_xpath('//*[@id="latestQuoteNewsStream-0- Stream"]/ul/li[9]/div/div/div[2]/h3/a').text)

Even if you do this it will work the same way since there is only one element with this id so simply use

#This should also work fine
print(driver.find_element_by_xpath('//*[@id="latestQuoteNewsStream-0- Stream"]').text)