Access Denied Page with selenium

Question

Im trying to code up a small application that uses footlocker and scrapes certain pages from the website. Now the way I want to do the app is by starting at the footlocker homepage and then clicking through different parts on the website. Below I have given an example of one of the additional links that I would click to then scrape. The issue though that I am having is that when the application finds the button and clicks on it I go to a error page kind of and then if I refresh the page I get an Access denied page. If anyone could help me with this issue I would greatly appreciate it. One idea that I thought might be the issue would have to deal with cookies but Im not experienced enough in web based applications to know if that is the case.

webpage = r"http://www.footlocker.com/" 
driver = webdriver.Chrome(r'C:\Users\saleh\Downloads\chromedriver_win32\chromedriver.exe')
driver.get(webpage)
driver.find_elements_by_xpath("//*[contains(text(), 'Sitemap')]")[0].click()

Answer 1

Try simulate what a normal browser would do:

add headers

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()

# Add headers
user_agent =  ('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) '
'AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/39.0.2171.95 Safari/537.36')
opts.add_argument(f'user-agent={user_agent}')

# Remove the Automation Info 
opts.add_argument('--disable-infobars')

# if you move chromedriver.exe into C:\Windows or C:\Users\saleh or location where this code is executed, then you don’t have to pass it here

chrome_exe = r'C:\Users\saleh\Downloads\chromedriver_win32\chromedriver.exe'

driver = webdriver.Chrome(chrome_exe, chrome_options=opts)

URL should not repr r

 webpage = 'http://www.footlocker.com/' driver.get(webpage)

Observer what you see. Open Developer Tools, and perform the step manually first while observer the elements you are interacting. Then write code to do the same steps.