I am trying to extract the text of the first listing from the url using Selenium and headless browser. The code works fine if it's not in headless mode, but not returning anything and not working fine in headless mode. Here's the code.
import requests
import time
from selenium import webdriver
from selenium.webdriver.firefox.options import Options as FirefoxOptions
user_agent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2'
options = FirefoxOptions()
options.add_argument(f'user-agent={user_agent}')
options.add_argument("--headless")
options.add_argument('window-size=1920x1080')
driver = webdriver.Firefox(options=options)
start_url = "https://ammoseek.com/ammo/300aac-blackout?pl=no&co=new&ca=brass" #works with US IP
driver.get(start_url)
time.sleep(5)
text = driver.find_element_by_class_name('results-card').text
print (text)
Try to run your code without specifying user-agent
.
You may also try to use Google Chrome webdriver instead.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.