Scrapy登录身份验证不起作用

Question

Hi I'm working with scrapy to log into some random website. I followed the tutorials on scrapy and it doesn't seem to be working. When i try it, i notice that the "isAuthenticated": False. And the html body i return doesn't contain everything that the actual website does. I'm not sure what the issue is. I thought it was the CSRFtoken but after research i found that scrapy should handle that. Here is the code below. Any Suggestions?

  import scrapy
  import sys
  from scrapy import Spider
  from scrapy import Request

  class IvanaSpider(Spider):
     name = 'ivanaSpider'

     def start_requests(self):
        return [scrapy.FormRequest(
              'https://bitbucket.org/account/signin/?next=/',
              formdata={'username': 'username', 'password': 'password',
                       'form_build_id': 'form - v14V92zFkSSVFSerfvWyH1WEUoxrV2khjfhAETJZydk',
                       'form_id': 'account_api_form',
                       'op': 'Sign in'
              },
              callback=self.after_login)]

     def after_login(self, response):
        # check login succeed before going on
        if "It's recommended that you log in" in response.body:
           print "------------------------------------------"
           self.logger.error("Login failed")
           return

        # continue scraping with authenticated session...
        for line in response.xpath('//body').extract():
           print line.encode(sys.stdout.encoding, errors='replace')

Answer 1

To login to website, you will need to use FormRequest but for some website, ie bitbucket in your example,

they use predefined form attributes like CSRFtoken, session info and other tokens which can only be used from the previous page user have visited

In such cases, one can use FormRequest.from_response method of scrapy which collect all pre-defined params from the response and post them as a formdata

# For example 
import scrapy
import sys
from scrapy import Spider
from scrapy import Request

class IvanaSpider(Spider):
    name = 'ivanaSpider'
    start_urls = (
        'https://bitbucket.org/account/signin/?next=/',
    )

    def parse(self, response):
        yield scrapy.FormRequest.from_response(
            response=response,
            formdata={"username": "<your username>",
                      "password": "<your password>"},
            #formname="login",apparently there are many socal login forms so select one based on xpath ( form id)
            formxpath=".//form[@id='aid-login-form']",
            callback=self.after_login,
            dont_click=True,

        )

    def after_login(self, response):
        # check login succeed before going on
        if "It's recommended that you log in" in response.body:
            print "------------------------------------------"
            self.logger.error("Login failed")
            return

        # continue scraping with authenticated session...
        for line in response.xpath('//body').extract():
            print line.encode(sys.stdout.encoding, errors='replace')