定时延迟或重定向后获取最终URL

Question

I am trying to scrape a website, but when I open the webpage it has 5 seconds redirect delay, ie you have to wait for 5 sec and then the real page loads. I have tried the following code .

from bs4 import BeautifulSoup
import time
import requests

r=requests.get("https://etherscan.io/address/0xc257274276a4e539741ca11b590b9447b26a8051",timeout=6)
time.sleep(5) 
print(r.history)

data=r.text

soup=BeautifulSoup(data)

print(soup.prettify())

But when I run the code I get the redirect page, not the final page. Thanks for help

Answer 1

It looks like etherscan.io is protected by Cloudflare , and Cloudflare is causing the delayed redirect that you are seeing. One of the purposes of Cloudflare is to prevent bots from making automated requests to the site (which seems a lot like what you are doing).

Getting around Cloudflare will not be easy. First, you'll need to make your requests 'look like' they are coming from a real browser - meaning that the tool that you are using to make these requests needs to present the same request headers that a real browser would, handle cookies like a browser would, run javascript like a browser would, etc.

Even if you succeed in doing all of the above, Cloudflare is likely to block your requests (or challenge them) after certain number of requests have been made over some period of time.

Answer 2

If you really really are set on using something other than selenium or the API (which would make the most sense), you could take a look at this . It's a scraper meant to handle cloudflare sites, but it requires some other things (most notably Node.js) to run. While this is pretty neat, seems like a pain when there are easier solutions.