Python selenium 如何从网站上抓取值列表

Question

I have a list of job roles on this website that I want to scrape. The code I am using is below:

driver.get('https://jobs.ubs.com/TGnewUI/Search/home/HomeWithPreLoad?partnerid=25008&siteid=5012&PageType=searchResults&SearchType=linkquery&LinkID=6017#keyWordSearch=&locationSearch=')
job_roles = driver.find_elements(By.XPATH, '/html/body/div[2]/div[2]/div[1]/div[6]/div[3]/div/div/div[5]/div[2]/div/div[1]/ul/li[1]/div[2]/div[1]/span/a')

for job_roles in job_roles:
    text = job_roles.text
    print(text)

With this code, I am able to retrieve the first role which is: Business Analyst - IB Credit Risk Change

I am unable to retrieve the other roles, can someone kindly assist

Thanks

Answer 1

In this case all the job names have the two CSS classes jobProperty and jobtitle .

So, since you want all the jobs, I recommend selecting by CSS selector .

The following example should work:

driver.get('https://jobs.ubs.com/TGnewUI/Search/home/HomeWithPreLoad?partnerid=25008&siteid=5012&PageType=searchResults&SearchType=linkquery&LinkID=6017#keyWordSearch=&locationSearch=')

job_roles = driver.find_elements_by_css_selector('.jobProperty.jobtitle')
for job_roles in job_roles:
    text = job_roles.text
    print(text)

Answer 2

If you want to use the xPath, you were very close. Your xPath specifically only selects the first li element ( li[1] ). By changing it to just li , it will find all matching xPaths:

driver.get('https://jobs.ubs.com/TGnewUI/Search/home/HomeWithPreLoad?partnerid=25008&siteid=5012&PageType=searchResults&SearchType=linkquery&LinkID=6017#keyWordSearch=&locationSearch=')
job_roles = driver.find_elements(By.XPATH, '/html/body/div[2]/div[2]/div[1]/div[6]/div[3]/div/div/div[5]/div[2]/div/div[1]/ul/li/div[2]/div[1]/span/a')

for job_roles in job_roles:
    text = job_roles.text
    print(text)