使用python flask上传大型csv文件的最佳方法

Question

Requirement : To Upload files using flask framework. Once uploaded to the server user should be able to see the file in UI.

Current code : In order to meet above requirement i wrote the code to upload sufficiently large files and its working fine with (~30 MB file, yes ofcourse not that fast). But when i am trying to upload (~100 MB) file, It is taking too long and process never completes.

This is what currently i am doing:

UPLOAD_FOLDER = '/tmp'

    file = request.files['filename']
    description = request.form['desc']

    filename = secure_filename(file.filename)
    try:
        file.save(os.path.join(UPLOAD_FOLDER, filename))
        filepath = os.path.join(UPLOAD_FOLDER, filename)
    except Exception as e:
        return e
    data = None
    try:
        with open(filepath) as file:
            data = file.read()
    except Exception as e:
        log.exception(e)

So what i am doing is first saving the file to temporary location in server and then from then reading the data and putting it into our database. I think this is where i am struggling i am not sure what is the best approach.

Should i take the input from user and return the success message( obviously user won't be able to access the file immediately then ) and make putting the data into database a background process, using some kind of queue system. Or What else should be done to optimize the code.

Answer 1

On the flask side make sure you have the MAX_CONTENT_LENGTH config value set high enough:

app.config['MAX_CONTENT_LENGTH'] = 100 * 1024 * 1024  # 100MB limit

Also you may want to look into the Flask-Upload extension .

There is another SO post similar to this one: Large file upload in Flask .

Other than that you problem may be a timeouts somewhere along the line. What does the rest of your stack look like? Apache? Nginx and Gunicorn? Are you getting a Connection reset error, Connection timed out error or does it just hang?

If you are using Nginx try setting proxy_read_timeout to a value high enough for the upload to finish. Apache may also have a default setting causing you trouble if that is what you are using. It's hard to tell without knowing more about your stack and what the error is that you are getting and what the logs are showing.