Scrapy Spider 无法登录 Discord

Question

i am trying to make a scraper for Discord to get all the members of a server, i am stuck at the login though, i can't find the csrf token anywhere in the source code for the page maybe that is why i'm getting this error since a few sources say that it is required but i'm not sure, here's my spider causing the problem

from scrapy.http import FormRequest

class RecruteSpider(scrapy.Spider):
    name = "Recruteur"

    def start_requests(self)
        urls = [
            'https://discord.com/login',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.login)

    def login(self, response):

        url = 'https://discord.com/login'
        formdata = {"username":"SecretUserName", "password":"SecretPassword"}
        yield FormRequest.from_response(
            response = response,
            url = url,
            formdata = formdata,
            callback = self.afterLogin
        )


    def afterLogin(self, response):
        print("Success!!")
        #do stuff

Wen i run the program i get the error

ValueError: No element found in <200 https://discord.com/login>

Even though there clearly is a form element at that url.

I have also tried using the login url as response variable in the Form response but i get the error

AttributeError: 'str' object has no attribute 'encoding'

if you need any extra detail feel free to ask, any help is greatly appreciated, thanks in advance.

Answer 1

The error you are getting is because discord loads the /login page using javascript and therefore the response does not contain any form elements. You need to render the javascript using either scrapy-playwright (personal favourite), selenium or scrapy-splash .

Also your formdata variable contains invalid keys. See screenshot of the payload sent to the server in the browser.

Using scrapy-playwright , I was able to get to the callback function as below. Also note that the discord server may require you to solve a captcha once you send the login request which presents another challenge that you will need to solve.

discord.py

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy import FormRequest

class RecruteSpider(scrapy.Spider):
    name = "Recruteur"

    def start_requests(self):
        urls = ['https://discord.com/login']
        for url in urls:
            yield scrapy.Request(url=url, callback=self.login, meta={"playwright": True})

    def login(self, response):
        url = 'https://discord.com/login'
        formdata = {"login":"SecretUserName", "password":"SecretPassword"}
        yield FormRequest.from_response(
            response = response,
            url = url,
            formdata = formdata,
            callback = self.afterLogin
        )

    def afterLogin(self, response):
        print("Success!!")
        #do stuff


if __name__ == "__main__":
    process = CrawlerProcess(settings={
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        }, })
    process.crawl(RecruteSpider)
    process.start()