如何仅使用内置模块使用Python 3上传大文件？

Question

I'm trying to upload a file 32+ MB to a server using an API. The only restriction I have is that can only use built-in modules. I have seen many examples using requests library, but I'm trying to solve with urllib . Using curl as PoC, I did the job like this:

curl -v --request POST --url 'https://domain/upload/long-string/' --form 'apikey=my-api-key' --form 'file=@my-file.extension'

Using urllib , I wrote the code below, but it doesn't work, because the server always returns a 400 error:

import urllib

def post_bigfile(upload_url, file, auth, timeout):
        headers = {'Accept': '*/*', 'Content-Type': 'multipart/form-data'}
        data = {'file': file, 'apikey': auth}
        req = urllib.request.Request(upload_url, headers=headers, 
            data=urlencode(data).encode('utf-8'), method='POST')
        return urllib.request.urlopen(req, timeout=timeout)

post_bigfile('https://domain/upload/long-string/', open('my-file.extension','rb'), 'my-api-key', 20)

I have tried using different values of Content-Type and Accept , but it still doesn't work. What could I possibly doing wrong? Is there another built-in module I could use to better solve this problem?

Answer 1

A hint to what you are doing wrong can be found here:

'Content-Type': 'multipart/form-data'
headers=headers

I was also busy with curl commands a while ago which were pushing multipart form-data and investigating doing it with built-in python libraries only (ie not requests). Once you put --form into curl you don't even need to give curl the --headers "Content-Type: multipart/form-data" argument, it just defaults to it. Maybe you didn't realise curl was doing that.

Have a look at this , python does not support that Mime type. A Python issue is linked https://bugs.python.org/issue3244 and I think your best bet is this script: https://pymotw.com/3/urllib.request/#uploading-files if you really want to do that with standard libraries only.

Am assuming you don't control the server which is receiving that file and it looks like it wants multipart/form-data with the two elements "apikey" and "file". Unless you can alter what that server wants to something which urllib can POST you will have to use requests or see if you can get that big MultiPartForm class to work.