Trying to select element by xpath with Selenium but getting error “Unable to locate element”

Question

I am trying to scrape the list of followings for a given instagram user. This requires using Selenium to navigate to the user's Instagram page and then clicking "following". However, I cannot seem to click the "following" button with Selenium.

driver = webdriver.Chrome()
url = 'https://www.instagram.com/beforeeesunrise/'
driver.get(url)
driver.find_element_by_xpath('//*[@id="react-root"]/section/main/article/header/div[2]/ul/li[3]/a').click()

However, this results in a NoSuchElementException . I copied the xpath from the html, tried using the class name, partial link and full link and cannot seem to get this to work! I've also made sure that the above xpath include the element with a "click" event listener.

UPDATE: By logging in I was able to get the above information. However (!), now I cannot get the resulting list of "followings". When I click on the button with the driver, the html does not include the information in the pop up dialog that you see on Instagram. My goal is to get all of the users that the given username is following.

Answer 1

Try a different XPath. I've verified this is unique on the page.

driver.find_element_by_xpath("//a[contains(.,'following')]")

Answer 2

Make sure you are using the correct X Path.

Use the following link to get perfect X Paths to access web elements and then try.

Selenium Command

Hope this helps to solve the problem!

Answer 3

It's not the main goal of selenium to provide rich functionalities, from a web-scraping perspective, to find elements on the page, so the better option is to delegate this task to a specific tool, like BeautifulSoup . After we find what we're looking for, then, we can ask for selenium to interact with the element.

The bridge between selenium and BeautifulSoup will be this amazing function below that I found here . The function gets a single BeautifulSoup element and generates a unique XPATH that we can use on selenium .

import os
import re
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import itertools

def xpath_soup(element):
    """
    Generate xpath of soup element
    :param element: bs4 text or node
    :return: xpath as string
    """
    components = []
    child = element if element.name else element.parent
    for parent in child.parents:
        """
        @type parent: bs4.element.Tag
        """
        previous = itertools.islice(parent.children, 0, parent.contents.index(child))
        xpath_tag = child.name
        xpath_index = sum(1 for i in previous if i.name == xpath_tag) + 1
        components.append(xpath_tag if xpath_index == 1 else '%s[%d]' % (xpath_tag, xpath_index))
        child = parent
    components.reverse()
    return '/%s' % '/'.join(components)


driver = webdriver.Chrome(executable_path=YOUR_CHROMEDRIVER_PATH)
driver.get(url = 'https://www.instagram.com/beforeeesunrise/')
source = driver.page_source

soup = bs(source, 'html.parser')
button = soup.find('button', text=re.compile(r'Follow'))
xpath_for_the_button = xpath_soup(button)

elm = driver.find_element_by_xpath(xpath_for_the_button)
elm.click()

...and works!

( but you need writing some code to log in with an account)