Find an element using href and click on it. Copying xpath does not work

Question

I am trying to scrape some information off this website using python's selenium.

First, I log into the website and get to the page . Then, I would like to click on the tab "Quickscan" to scrape some info. However, that's where I get stuck. I can't find a way to click on the tab.

Note that this problem would be surmounted if I managed to navigate to the page , though when I log in, even if I put such page in my WebDriver, I still get redirected to this one .

To get to the desired page I have tried finding the element both through the xpath and through the link, but it does not find the element.

import requests
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

driver =webdriver.Chrome(executable_path ="mypath")

driver.get("https://vc4a.com/ventures/autocollect/#quickscan-tab")



#find username and password bar
username = driver.find_element_by_id("user_login")
password = driver.find_element_by_id("user_pass")

#Input password and username
username.send_keys("username")
password.send_keys("password")

#click on submit
driver.find_element_by_name("wp-submit").click()
driver.find_element_by_name("rememberme").click()

#try to find element using text in the link
driver.find_elements_by_link_text('#quickscan-tab')[0].click()

#try to find element using xpath from the inspected element
driver.find_element_by_xpath('//*[@id="subnav"]/li[3]/a').click()

I would like to be able to open the tab so that I can scrape the content.

When I use the first code it returns the following error:

IndexError: list index out of range

However, by inspecting the page I can see there is indeed 2 elements with the text "#quickscan-tab", so I don't understand why the index 0 would be out of range.

When I use the second code it returns the following error:

NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="subnav"]/li[3]/a"}
  (Session info: chrome=74.0.3729.169)
  (Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Mac OS X 10.14.5 x86_64)

What I did was just copying the xpath.

Answer 1

driver.find_elements_by_link_text('#quickscan-tab')[0].click() - it's wrong

Link text doesn't work like this you need to create a different locator. try below XPath

driver.find_element_by_xpath((//*[@id='quickscan-tab'])[0])

Answer 2

I created an account on that page and tried this modified script and it works:

import requests
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

driver = webdriver.Chrome()

driver.get("https://vc4a.com/ventures/autocollect/#quickscan-tab")

#find username and password bar
username = driver.find_element_by_id("user_login")
password = driver.find_element_by_id("user_pass")

#Input password and username
username.send_keys("username")
password.send_keys("password")

#click on submit
driver.find_element_by_name("rememberme").click()
driver.find_element_by_name("wp-submit").click()
time.sleep(10)

#try to find element using text in the link
driver.find_elements_by_link_text('Quickscan')[0].click()

#try to find element using xpath from the inspected element
driver.find_element_by_xpath('//a[text()="Quickscan"]').click()

1. link_text means the text that you actually see. [Quickscan]
2. Login takes time and the script tries to locate before the tab is created thus causing error.
3. Your xpath would have worked if not for the login delay.
4. Click rememberme before submitting the form. Or don't, since selenium starts a clean session for every run.

Answer 3

Try this:

scanelements = driver.find_elements_by_xpath('//*[@id='quickscan-tab']')
for elt in scanelements :
   elt.click()
   break