从金字塔FileResponse下载NamedTemporaryFile是否安全？

Question

I'm currently working on an export feature for a web application using Pyramid on Python and running on Ubuntu 14.04. It zips the files into a NamedTemporaryFile and sends it back through a FileResponse:

# Create the temporary file to store the zip
with NamedTemporaryFile(delete=True) as output:
    map_zip = zipfile.ZipFile(output, 'w', zipfile.ZIP_DEFLATED)
    length_mapdir = len(map_directory)

    for root, dirs, files in os.walk(map_directory, followlinks=True):
        for file in files:
            file_path = os.path.join(root, file)
            map_zip.write(file_path, file_path[length_mapdir:])

    map_zip.close()

    #Send the response as an attachement to let the user download the file
    response = FileResponse(os.path.abspath(output.name))
    response.headers['Content-Type'] = 'application/download'
    response.headers['Content-Disposition'] = 'attachement; filename="'+filename+'"'
    return response

On the client's side, the export takes some time then the file download popup appears, nothing goes wrong and everything is in the zip as planned.

While the file is zipping, I can see a file taking up more and more size in /tmp/, and before the download popup appears, the file disappears. I assume this is the NamedTemporaryFile.

While the file is being zipped or downloaded, there isn't any significant change in the amount of RAM being used, it stays around 40mb while the actual zip is over 800mb.

Where is pyramid downloading the file from? From what I understand of tempfile, it is unlinked when it is closed. If that's true, is it possible another process could write on the memory where the file was stored, corrupting whatever pyramid is downloading?

Answer 1

In Unix environments something called reference counting is used when a file is created, and opened. For each open() call on a file, the reference number is increased, for each close() it is decreased. unlink() is special in that when that is called the file is unlinked from the directory tree, but will remain on disk so long as the reference count stays above 0.

In your case NamedTemporaryFile() creates a file on disk named /tmp/somefile

1. /tmp/somefile now has a link count of 1
2. /tmp/somefile then has open() called on it, so that it can return the file to you, this increases the reference count to 1
3. /tmp/somefile is then written to by your code, in this case a zip file
4. /tmp/somefile is then passed to FileResponse() which then has open() called on it, increasing the reference count to 2
5. You exit the scope of the with statement, and NamedTemporaryFile() calls close() followed by unlink() . Your file now has 1 reference to it, and a link count of 0. Due to the reference still existing, the file still exists on disk, but can no longer be seen when searching for it.
6. FileResponse() is iterated over by your WSGI server, and eventually once the file has been fully read, your WSGI server calls close() on it, dropping the reference count to 0, at which point the file system will clean the file up entirely

It is at that last point that the file is no longer accessible. In the mean time your file is completely safe and there is no way for it to be overwritten in memory or otherwise.

That being said, if FileResponse() was lazy loaded for example (ie it wouldn't open() the file until the WSGI server started sending the response), it would be entirely possible that it would attempt to open() the temporary file too late, and NamedTemporaryFile() would have already deleted the file. Just something to keep in mind.