使用 python 抓取 Indeed web 时遇到 403 错误

Question

I need to do the web scraping using request in 'https://in.indeed.com/'. When I'm running the code it shows the 403 error

Can anyone tell me the solution..

url = "https://in.indeed.com"

hdr = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'}

result = requests.get(url,headers = hdr)

print(result)

I have tried till this to check the status code of the website only it shows error

Note: Need to do the web scraping without using selenium

Answer 1

It appears that there are some headers missing in your request. I also get a 403 when i do the request like that. However, a copied cURL request works:

Try the following:

1. Open the website in Chrome (or Firefox)
2. Open the developers console
3. Copy the request from the first GET (4. You can try out the cURL request in your console. It worked for me without any 403)
4. Set the headers similar to the cURL request in your code

(However, i'm assuming the take some measures against web scraping. So you may run in further problems. I'm guessing that you also have to save the cookie or something like this.)