如何使用 selenium 提取 href 链接标题？

Question

我在弄清楚如何使用 selenium 提取页面上的所有 100 个链接标题时遇到问题。 我有只提取链接的代码，但是除了链接标题之外，我将如何 go 这样做呢？ 任何帮助将不胜感激，谢谢！

我有的：

from selenium import webdriver
driver = webdriver.Chrome('/Users/wwds/Downloads/chromedriver')
driver.get('https://www150.statcan.gc.ca/n1/en/type/data?count=100')

elems = driver.find_elements_by_css_selector(".ndm-result-title [href]")
links = [elem.get_attribute('href') for elem in elems]

Answer 1

从一个页面上的所有100个链接中获取title 。你需要诱导WebDriverWait ()并等待visibility_of_all_elements_located ()和下面的css选择器。

driver.get('https://www150.statcan.gc.ca/n1/en/type/data?count=100')
elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "#all a[target='_self']")))
linkTitles = [title.text for title in elements]
print(len(linkTitles))
print(linkTitles)

您需要导入以下库。

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Output 控制台：

100
['Capital markets activities, gross new issues, retirements and net new issues, par value, Bank of Canada', 'Funds advanced, outstanding balances, and interest rates for new and existing lending, Bank of Canada', 'Funds advanced, outstanding balances, and interest rates for new and existing lending, dollars, Bank of Canada', 'Net new issues of bonds and equities by corporations and federal government enterprises: Industrial classification, Bank of Canada', 'Bank of Canada, assets and liabilities, average of Wednesdays', 'Chartered banks, assets and liabilities, monthly average, Bank of Canada', 'Currency outside banks and chartered bank deposits, monthly average, Bank of Canada', 'Selected credit measures, monthly average or average of month-ends, unadjusted unless otherwise noted, Bank of Canada', 'Chartered bank assets and liabilities and monetary aggregates, monthly average, seasonally adjusted, Bank of Canada', 'Consumer credit, outstanding balances of selected holders, Bank of Canada', 'Credit measures, Bank of Canada', 'Corporate short-term paper outstanding, Bank of Canada, monthly', 'Toronto Stock Exchange statistics', 'Residential mortgage credit, outstanding balances of major private institutional lenders, Bank of Canada', 'Bonds outstanding, shown at par value, by currency of payments and issuers, Bank of Canada', 'Bank of Canada, assets and liabilities, Wednesdays', 'Bank of Canada, money market and other interest rates', 'Positions of members of the Canadian Payments Association and Bank of Canada operations, Bank of Canada, weekly', 'Chartered banks, classification of non-mortgage loans, at end of period, Bank of Canada', 'Tobacco, sales and inventories, monthly production', 'Historical (real-time) releases of Consumer Price Index (CPI) statistics, measures of core inflation - Bank of Canada definitions', 'Historical (real-time) releases of wholesale sales, price and volume, seasonally adjusted', 'Historical (real-time) releases of wholesale trade, sales', 'Historical (real-time) releases of wholesale trade, inventories', 'Historical (real-time) releases of retail trade, sales', 'Historical (real-time) releases of retail sales, price, and volume', 'Capital expenditures, oil and gas extraction industries, Canada', 'Daily average foreign exchange rates in Canadian dollars, Bank of Canada', 'Investment in Building Construction', 'Gross domestic product (GDP) at basic prices, by industry, provinces and territories, percentage share', 'Gross domestic product (GDP) at basic prices, by industry, provinces and territories', 'Gross domestic product (GDP) at basic prices, by industry, provinces and territories, growth rates', 'Canadian Economic Dashboard and COVID-19', 'Statistics Canada - Data Visualization Products', 'Financial market statistics, last Wednesday unless otherwise stated, Bank of Canada', 'Selected United States dollar interest rates, Wednesdays, Bank of Canada', 'Fisher commodity price index, United States dollar terms, Bank of Canada, Fisher commodity price index, United States dollar terms, Bank of Canada, weekly', 'Financial market statistics, as at Wednesday, Bank of Canada', 'Exports and imports of goods and services, quarterly, Canada, (NAPCS 2017)', 'Contributions to annual percent change in real exports and imports of goods and services, Canada, annual, (NAPCS 2017)', 'Industrial product price index, by major product group, monthly', 'Industrial product price index, by major product group, percentage change, monthly', 'Industrial product price index, by product, monthly', 'Industrial product price index, by product, percentage change, monthly', 'Industrial product price index, for selected products, by region, monthly', 'Industrial product price index, for selected products, by region, percentage change, monthly', 'Industrial product price index, by industry, monthly', 'Industrial product price index, by industry, percentage change, monthly', 'Raw materials price index, monthly', 'Raw materials price index, percentage change, monthly', 'Commercial software price index, monthly', 'Commercial software price index, percentage change, monthly', 'Couriers and messengers services price index, monthly', 'Couriers and messengers services price index, percentage change, monthly', 'Computer price index, by type of purchaser, monthly', 'Computer price index, by type of purchaser, percentage change, monthly', 'Computer peripheral price index, monthly', 'Computer peripheral price index, percentage change, monthly', 'Smart phone price index, monthly', 'Smart phone price index, percentage change, monthly', 'Civil aviation operating statistics, by type of service, Canadian air carriers, Levels I and II, quarterly', 'Civil aviation operating statistics, by sector, Canadian air carriers, Levels I and II, quarterly', 'Civil aviation financial statistics, Canadian air carriers, Levels I and II, quarterly', 'Civil aviation financial performance indicators, Canadian air carriers, Levels I and II, quarterly', 'Civil aviation productivity measures, Canadian air carriers, Levels I and II, quarterly', 'Weekly rail terminal performance indicator, Transport Canada', 'Weekly performance indicators of grain transportation by rail, Transport Canada', 'Weekly rail system performance indicators, by commodities, Transport Canada', 'Weekly rail system performance indicator, by type of rail car, Transport Canada', 'Flows and stocks of fixed non-residential and residential capital, by sector and asset, provincial and territorial', 'Gross domestic product, income-based, quarterly', 'Gross domestic product, expenditure-based, Canada, quarterly', 'Gross national income and gross domestic income, indexes and related statistics, quarterly', 'Gross domestic product price indexes, quarterly', 'Household final consumption expenditure, quarterly, Canada', 'Gross fixed capital formation, quarterly, Canada', 'Investment in inventories, quarterly, Canada', 'Current and capital accounts - National, Canada, quarterly', 'Current and capital accounts - Households, Canada, quarterly', 'Compensation of employees, Canada, quarterly', 'Current and capital accounts - Non-profit institutions serving households, Canada, quarterly', 'Current and capital accounts - Corporations, Canada, quarterly', 'Undistributed corporation profits', 'Current and capital accounts - General governments, Canada, quarterly', 'Current and capital accounts - Non-residents, Canada, quarterly', 'Gross domestic income, gross national income and net national income, Canada, quarterly', 'Gross domestic product, expenditure-based, at 2012 constant prices, quarterly', 'Detailed household final consumption expenditure, Canada, quarterly', 'Corporation profits before taxes, on an original-cost-basis, excluding government business enterprises, Canada, quarterly', 'Property income of households, Canada', 'General governments final consumption expenditure, Canada, quarterly', 'Contributions to annual percent change in real expenditure-based gross domestic product, Canada, annual', 'Gross national income and gross domestic income, indexes and related statistics, annual', 'Gross domestic product price indexes, annual (2012=100)', 'Contributions to annual percent change in real household final consumption expenditure, Canada, annual', 'Contributions to annual percent change in real gross fixed capital formation, Canada, annual', 'Contributions to annual percent change in real general governments final consumption expenditure, Canada, annual', "Wages, salaries and employers' social contributions", 'Gross domestic product, expenditure-based, at 2012 constant prices, annual', 'Gross domestic product (GDP) at basic prices, by industry, monthly']

Answer 2

尝试使用xpath查找元素，并通过.text获取标题，如下所示：

driver.get('https://www150.statcan.gc.ca/n1/en/type/data?count=100')
elems = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//details[@id='all']//a[@target]")))
links = [elem.text for elem in elems]

导入后：

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC