无法使用请求登录网站

Question

I'm trying to log in to this site using requests module but I get 403 status code every time I try with my following attempt. Although I tried to mimic the way the requests is being sent by monitoring dev tools, I can't make it work. The credentials (username: simpndev@gmail.com , password: +agb5E2?w2pQJ3z ) that I've used here are for test purposes only, so you are free to use.

To get the form, all you need to do is click on the login button and then on the Fantasy button.

I've tried with:

import re
import requests

link = 'https://www.fanduel.com/contests'
url = 'https://api.fanduel.com/sessions'

payload = {"email":"simpndev@gmail.com","password":"+agb5E2?w2pQJ3z","product":"DFS"}

def log_in(s):
    r = s.get(link)
    client_id = re.findall(r"clientId\":\"(.*?)\",",r.text)[0]
    s.headers['authorization'] = f'Basic {client_id}'
    s.headers['Referer'] = 'https://www.fanduel.com/login'
    s.headers['accept'] = 'application/json'
    r = s.post(url,json=payload)
    print(r.status_code)

if __name__ == '__main__':
    with requests.Session() as s:
        s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36'
        log_in(s)

I found success using selenium, so I don't wish to go that route.

How can I log in to that site using requests?

Answer 1

There are 2 potential 403 errors involved in this request. The 403 from bot protection at link = 'https://www.fanduel.com/contests' and a 403 from the request to /sessions/

The 403 from link is using some advanced features of browsers to check for repeated attempts to login. That's going to be a more complicated subject involving a comparison of User-Agent strings to HTTP2 traffic, beating captchas with ML and so forth. My advice is not to go down that path.

Instead, lower the version of your user-agent string and make sure you're providing the correct headers. You're not providing the correct headers to the initial request and so you're generating 403s and then you're on a bot blacklist and trying to manage that.

The following works for me as shown in the debugger screenshot below:

import re
import requests

link = 'https://www.fanduel.com/contests'
url = 'https://api.fanduel.com/sessions'

payload = {"email":"simpndev@gmail.com","password":"+agb5E2?w2pQJ3z","product":"DFS"}

def log_in(s):
    r = s.get(link)
    client_id = re.findall(r"clientId\":\"(.*?)\",",r.text)[0]
    s.headers['Authorization'] = f'Basic {client_id}'
    s.headers['Referer'] = 'https://www.fanduel.com/login?cc_success_url=%2Fcontests'
    s.headers['Accept'] = 'application/json'
    s.headers['Accept-Encoding'] = 'gzip, deflate, br'
    s.headers['Accept-Language'] = 'en-US,en;q=0.5'
    s.headers['Origin'] = "https://www.fanduel.com"
    r = s.post(url,json=payload)
    print(r.status_code)

if __name__ == '__main__':
    with requests.Session() as s:
        s.headers['User-Agent'] = 'Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14 wimb_monitor.py/1.0'
        log_in(s)

Notice I modified your Referer, modified capitalization on your header keys, and provided what might be thought of as superfluous headers. When checking r.request.headers I saw differences between those being sent by requests and eg Firefox, so I just added in anything that was different.

Also notice that your bearer token and account credentials are now spread far and wide and may be contributing to additional 403s if you're still using those to test with. You'll want a clean account since many people may have these creds now.

Answer 2

Been there, had lot of issues using Selenium in the past for web scraping.

One alternative that I've got more into is using mitmproxy to dump the navigation script to a file and replaying it using requests , something like this:

import mitmproxy.io
import requests

def main():
    sess = requests.Session()
    with open('flows', 'rb') as f: data = [row for row in mitmproxy.io.FlowReader(f).stream()]
    for d in data:
        url = d.request.url
        raw_content = d.request.raw_content
        headers = dict([(k, v,) for k, v in d.request.headers.items() if not k.startswith(':') and k.lower() != 'cookie'])
        print(url, raw_content, headers)
        if d.request.method == 'GET':
            r = sess.get(url, headers=headers, data=raw_content)
        if d.request.method == 'POST':
            r = sess.post(url, headers=headers, data=raw_content)
        print(r.status_code)
        if d.request.url == 'https://api.fanduel.com/sessions':
            break

if __name__ == '__main__':
    main()

The code above (using my own flow dumped from mitmproxy) results in the output below:

http://www.fanduel.com/ b'' {'Host': 'www.fanduel.com', 'Upgrade-Insecure-Requests': '1', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Mobile/15E148 Safari/604.1', 'Accept-Language': 'en-gb', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive'}
200
https://www.fanduel.com/ b'' {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Mobile/15E148 Safari/604.1', 'accept-language': 'en-gb', 'accept-encoding': 'gzip, deflate'}
200
https://www.fanduel.com/JMCVuBG8/init.js b'' {'accept': '*/*', 'accept-encoding': 'gzip, deflate, br', 'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Mobile/15E148 Safari/604.1', 'accept-language': 'en-gb', 'referer': 'https://www.fanduel.com/'}
200
https://www.fanduel.com/login?source=Header%20Login b'' {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'accept-encoding': 'gzip, deflate, br', 'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Mobile/15E148 Safari/604.1', 'accept-language': 'en-gb', 'referer': 'https://www.fanduel.com/'}
200 
https://api.fanduel.com/sessions b'{"email":"simpndev@gmail.com","password":"+agb5E2?w2pQJ3z","product":"DFS"}' {'accept': 'application/json', 'origin': 'https://www.fanduel.com', 'content-type': 'application/json', 'authorization': 'Basic ZWFmNzdmMTI3ZWEwMDNkNGUyNzVhM2VkMDdkNmY1Mjc6', 'referer': 'https://www.fanduel.com/login', 'content-length': '75', 'accept-language': 'en-gb', 'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Mobile/15E148 Safari/604.1', 'accept-encoding': 'gzip, deflate, br'}
201

All you'll need is a device with a proxy set to pass through your own installation of mitmproxy (I usually use my phone to route the requests using my PC and SOCKS5). If something changes in the navigation of the website, all you'll need to do is build a new flow and dump the script to feed the python program.

That won't solve more complex javascript security implementations, but should be enough for simple ones such as this website's.