Python POST 请求返回 404 状态代码，但 GET 请求返回 200

Question

I am trying to login to this website. When I submit a simple GET request I get a 200 response as expected.

import requests

login_url = 'https://urs.earthdata.nasa.gov/'

sess = requests.Session()
sess.headers ={'User-Agent':'Mozilla/5.0'}
sess.verify=False
r1 = sess.get(login_url)
print (r1.status_code)
>>>200

However when I try to actually log in using a POST request I get a 404 error. (The payload for the request has been sourced from the html for the page and using page inspectors in Firefox.)

import requests
import re

product_url = 'https://datapool.asf.alaska.edu/L1.5/A3/ALPSRP171431190-L1.5.zip'
login_url = 'https://urs.earthdata.nasa.gov/'
username = 'username'
password = 'password'

sess = requests.Session()
sess.headers ={'User-Agent':'Mozilla/5.0'}
sess.verify=False

r1 = sess.get(product_url)

tkn_ptn = '<meta name="csrf-token" content="(.*==)" />'
tkn = re.search(tkn_ptn,r1.text).group(1)
print('CSRF Token: {}'.format(tkn))
>>>'CSRF Token: CDOX5tOhBtX2vvZn/c/MLRaYJtW7hzeQLm/eEVn09cHosnlsR/5P8a+k4YEaAzYQZRxCgNf9evDqyhWiZiefmQ=='

cli_ptn = '<input type="hidden" name="client_id" id="client_id" value="(.*)" />'
cli = re.search(cli_ptn,r1.text).group(1)
print('Client ID: {}'.format(cli))
>>>'Client ID: BO_n7nTIlMljdvU6kRRB3g'

redir_ptn = '<input type="hidden" name="redirect_uri" id="redirect_uri" value="(.*?)" />'
redir = re.search(redir_ptn,r1.text).group(1)
print ('Redirect URL: {}'.format(redir))
>>>'Redirect URL: https://auth.asf.alaska.edu/login'

payload = {'username':username, 
            'password':password,
            'authenticity_token' : tkn,
            'client_id' : cli, 
            'redirect_uri': redir,
            'response_type' : 'code', 
            'stay_in' : '1', 
            'commit':'Log in'}

r2 = sess.post(login_url, data=payload)
print (r2.status_code)
>>>404

Why won't the page accept my payload and let me login?

Answer 1

The payload data comes from the login page itself, so that would be https://urs.earthdata.nasa.gov/ looking at network tab on your browser.
I just entered some random username and password and looking at my network tab i see a POST being made to https://urs.earthdata.nasa.gov/login . Looking at the payload, this is the format it has:

utf8: ✓
authenticity_token: ...token base64...
username: 123
password: 123
client_id: 
redirect_uri: 
commit: Log in

So we just need to extract the authenticity_token from the source. looking at the source for the login page we see this bit:

<form id="login" action="/login" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="&#x2713;" /><input type="hidden" name="authenticity_token" value="...token base64..." />

So we just use some regex to extract this bit as it's faster for single-use stuff like this(you could use any other method or regex form you want):

token = re.search(r'authenticity_token".*?"(.*?)"', webpage.text).group(1)

and finally creating the data and passing it to the POST method:

data = {
"utf8": "✓",
"authenticity_token": token,
"username": username,
"password": password,
"client_id": "",
"redirect_uri": "",
"commit": "Log in",
}
login = requests.post("https://urs.earthdata.nasa.gov/login", headers={'User-Agent':'Mozilla/5.0'}, data=data)