selenium phantomjs无法刮取网站机器人检测

Question

I can't scrape this site here is a screenshot of the request on python selenium phantomjs. I dont know how they detected it was a bot but says on the picture need javascript and need captcha and maybe what other things needed to? Definitely Im not scraping at superhuman speed because it is my first request so it was not the cause. PS when I paste the same request on my browser it directs to the page that i want and works okay.

    br = webdriver.PhantomJS('bin/phantomjs')
    br.set_window_size(1366, 200)
    br.get("website")
    br.save_screenshot(x)

Answer 1

Well I got it working now. I'll simply put this for the sake of other people who doesn't. enable javascript and fake useragent

    cap = webdriver.DesiredCapabilities.PHANTOMJS
    cap["phantomjs.page.settings.javascriptEnabled"] = True
    cap["phantomjs.page.settings.loadImages"] = True
    cap["phantomjs.page.settings.userAgent"] = 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'
    br = webdriver.PhantomJS('bin/phantomjs',desired_capabilities=cap)

Answer 2

Things that can help in general :

• Headers should be similar to common browsers, including :
  • User-Agent : use a recent one (see https://developers.whatismybrowser.com/useragents/explore/ ), or better, use a random recent one if you make multiple requests (see https://github.com/skratchdot/random-useragent )
  • Accept-Language : something like " en,en-US;q=0,5 " (adapt for your language)
  • Accept : a standard one would be like " text/html,application/xhtml+xml,application/xml;q=0.9, / ;q=0.8 "
• Navigation :
  • If you make multiple request, put a random timeout between them
  • If you open links found in a page, set the Referer header accordingly
  • Or better, simulate mouse activity to move, click and follow link
• Images should be enabled
• Javascript should be enabled
  • Check that " navigator.plugins " and " navigator.language " are set in the client javascript page context
  • Check that the client you use does not inject noticeable javascript variables (like _cdc, __nightmare...)
• Use proxies