I am new in python selenium. and i want to get all the hidden href link
<div class="page-body">
<div class="page-title"></div>
<div class="page cursorPointer">
<a title="" data-placement="top" data-toggle="tooltip" href="#" data-original-title="Verified"></a></div>
</div>
and this my script:
#!/usr/bin/python3
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.chrome.options import Options
import requests
import re
from openpyxl import Workbook
driver = webdriver.Chrome(options=options)
driver.get(
"https://someurl.com")
pagelist = []
content = driver.page_source
soup = BeautifulSoup(content, 'lxml')
for a in soup.findAll('div', attrs={'class': 'page cursorPointer'}):
page = a.find_element_by_xpath("//a[@href]")
pagelist.append(page.get_attribute("href"))
df = pd.DataFrame({'Page': pagelist})
df.to_excel('pagelist.xlsx', index=False, encoding='utf-8')
i got this error:
page = a.find_element_by_xpath("//a[@href]") TypeError: 'NoneType' object is not callable
That happened because you're using a selenium method on a soup object. Try it like this:
pagelist = driver.execute_script("""
return [...document.querySelectorAll('a[href]')].map(a => a.href)
""")
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.