POST请求包括使用python的文件

Question

I am a beginner in web programming, so sorry if this is a very basic thing, but couldn't find something as specific as the problem that I have in stackoverflow. So I have a lot of text files (10k) that I need to upload to this website https://rostlab.org/services/nlsdb/ and then click on "Evaluate NES/NLS". This triggers a SQL query and returns me some info in table form back. I then need to click "CSV" button to get the file downloaded into my computer. Of course I don't want to upload each file manually, so I was trying to generate the requests in Python but could't get it done, I didn't even get to the point of have the table response from the initial website, so downloading the CSV is a challenge that I haven't met yet:

import requests

url = 'https://rostlab.org/services/nlsdb/query'
files = {'file-upload': ('some.txt', open('C:\\some.txt', 'rb'), 'text/plain')}
data = {'_token':'', 'input-data':'', 'query-sig2':''}

r = requests.post('https://rostlab.org/services/nlsdb/query', files=files, data=data)

As response, I am getting a shitload of text which I can resume in an error code 500 from HTML, so I am definetly doing something wrong here I can't see what. The POST request from the website when I submit the file looks something like this:

**General**
  Request URL:https://rostlab.org/services/nlsdb/query
  Request Method:POST
  Status Code:200 OK
  Remote Address:131.159.28.73:443
  Referrer Policy:no-referrer-when-downgrade

Response Headers
  Cache-Control:no-cache, private
  Connection:Keep-Alive
  Content-Encoding:gzip
  Content-Length:2231
  Content-Type:text/html; charset=UTF-8
  Date:Thu, 08 Feb 2018 12:39:30 GMT
  Keep-Alive:timeout=5, max=100
  Server:Apache
  Set-Cookie:nlsdb_session=eyJpdiI6IjZMRk03ZjRCNjBmU1JcL3Y0Vko4ZHFRPT0iLCJ2YWx1ZSI6Ikh2bHcyZHBuN25nNmx1QnRoOFlPMWhWU0RYdUpEdnAwbGtySWgwbDlDVElHZmRyNlBMeEdXT3ROSERcLzRRNDB2ZnVUQ2oyTDlmOVRHa3JNUUZJTnBkUT09IiwibWFjIjoiZWM3ZjFjYmQ2ZThkNmRlM2JmOTY5OWZiYWMxOTA4ZmZiZjcxZjU1ODJjNjU1ODgzYjczMmUxMGY1NGMwMjNlMCJ9; expires=Thu, 08-Feb-2018 14:39:30 GMT; Max-Age=7200; path=/; httponly
  Set-Cookie:XSRF-TOKEN=eyJpdiI6IjExMjBaRHNmWHVLZTBzSURYZFwvUmF3PT0iLCJ2YWx1ZSI6InQyWUE5QzZEd2xmZU5rMjlyekV1Z2JcL3lGNkNvbHl1TnBHMVh5eWtLeWtNb3JHcTJJSFpyR0lDVkxNV2h2cGsrTUhYMGl3ZDBET0hucHdpNzV0YkRpdz09IiwibWFjIjoiNzcxODBhYjIzYjEzNDU1OTNhNGRhNjI3OTAxNWY1MjFkYjI5MWQ5NjgwNGE4ZjVmMzQzZThkNWUzZWE0YTgwYSJ9; expires=Thu, 08-Feb-2018 14:39:30 GMT; Max-Age=7200; path=/
  Vary:Accept-Encoding

Request Headers
 Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
  Accept-Encoding:gzip, deflate, br
  Accept-Language:en-US,en;q=0.9
  Cache-Control:max-age=0
  Connection:keep-alive
  Content-Length:1943
  Content-Type:multipart/form-data; boundary=----WebKitFormBoundary1tOuJdyWl1bn7H4X
  Cookie:XSRF-TOKEN=eyJpdiI6IjZaWHdTa3FPYmNHbkxsNVpoUlE3T0E9PSIsInZhbHVlIjoiQWMraGlLekd1akkrc0RDTzNMRGNIcVFkVGdBNjZFa2h4XC8xcUI0VmtIVG9CTnVPNW1IUW55NU9iNGlGY0NCWkFkd0hDZnJOaXBaT3J0VHZTSXl6b1FBPT0iLCJtYWMiOiJmMjE3N2JkZDIyMjRkNTY3ZGE4MDhlNGY5OWJiMDAwYjNiNzYyNGJjMTc2YzA4NTQwODcxZTM3YjI0YjQ5MWUyIn0%3D; nlsdb_session=eyJpdiI6IjByb2dtS0Q1ekFBU1F0WURJUk8rWnc9PSIsInZhbHVlIjoiM3lMNFU5Y2hBXC9BVU0xT0RUNnhVaUJ0ckJ0RnB5QlJqbk15alNSNkM4MjhNTGd6TFwvR0dwd0ZpWE9pU3piekhWb3ZzQjNZYVQ4ODdHeUxUMVJWM0pwUT09IiwibWFjIjoiYTE1Y2Q2NmRlN2M4Yjc1MzEyZTQxYjcwMzVmYjNiNjA1YjdiNjU4ODkxZWJhM2JmYTAwYTk1MWNhZWNkNTczMiJ9
  DNT:1
  Host:rostlab.org
  Origin:https://rostlab.org
  Referer:https://rostlab.org/services/nlsdb/
  Upgrade-Insecure-Requests:1
  User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36

Request Payload  
  ------WebKitFormBoundary1tOuJdyWl1bn7H4X
  Content-Disposition: form-data; name="_token"

  GnjGT2Ejrrpo4Nlf2EbwtmLtY29GNFnoTJpl5z5o
  ------WebKitFormBoundary1tOuJdyWl1bn7H4X
  Content-Disposition: form-data; name="input-data"


  ------WebKitFormBoundary1tOuJdyWl1bn7H4X
  Content-Disposition: form-data; name="file-upload"; filename="some.txt"
  Content-Type: text/plain


  ------WebKitFormBoundary1tOuJdyWl1bn7H4X
  Content-Disposition: form-data; name="query-sig2"

  sF4MZkIaMc1K9TPZ6uYJuQ
  ------WebKitFormBoundary1tOuJdyWl1bn7H4X--

I believe that the data object is not correct, but I was not able to make it right, and omitting it does not seem to work neither. Any suggestion on how to retrieve the data correctly, and then downloading the corresponding csv file?

Answer 1

The site uses cross-site scripting tokens to protect against a common class of attacks. Furthermore, they use generated tokens for their submit buttons as well.

To be able to post anything, you need to:

• Store and return cookies. This is easiest with a session object
• Load the form page and read out the CSRF token and submit button values
• Use the extracted tokens in your POST request

I'd use BeautifulSoup to parse the form page and extract the tokens:

from bs4 import BeautifulSoup
import requests

form_url = 'https://rostlab.org/services/nlsdb/'

with requests.session() as sess:
    response = sess.get(form_url)
    soup = BeautifulSoup(response.content, 'html.parser')

    csrf_token = soup.find('input', {'name': '_token'})['value']
    submit_token = soup.find('button', id='submit-sig2')['value']        
    action_url = soup.find('form', id='input-form')['action']

    data = {'_token': csrf_token, 'query-sig2': submit_token, 'input-data':''}

    with open('C:\\some.txt', 'rb') as some_text:
        files = {'file-upload': ('some.txt', some_text, 'text/plain')}
        response = sess.post(action_url, data=data, files=files)

Note that I also extract the action attribute of the form tag; best to stick to what the server tells us to use.

The above code produces a 200 OK response with a HTML page listing matching results in a table.