By passing incapsula security- python

Question

I am trying to crawl the football data from the whoscored.com, the website has incapsula web oriented security which is not letting me crawl. Initially, I tried to give user_agent and changed the header then it worked but that's only for the first page. As I need to crawl some other parts of the website it keeps blocking me to request the website. It's getting exhausting now since the blocking time has been increasing.

Is there anybody who could suggest something to bypass the incapsula security mechanisms.

I need data for study purposes.

I have gone through all the old the previous question asked about this topic. but it does not help.

Tools. anaconda, language python, and library beautiful soup

Thanks.

Answer 1

if you mimic browser headers and appropiate time between requests, it will probably work

look at your request headers and that of your browser