Copying zipped file using python

Question

I need to copy file from my NAS to my local folder, zip it and then copy it elsewhere. This is my code:

from shutil import copyfile
import zipfile

def make_done_file(user, filename):
    copyfile(network_location+filename+'.csv', test_location+filename+'.csv')
    zip_file = zipfile.ZipFile(test_location+filename+'.csv.zip', 'w')
    zip_file.write(test_location+filename+'.csv', compress_type= None, arcname = filename+'.csv')
    copyfile(test_location+filename+'.done' , final_location+filename+'.done')
    copyfile(test_location+filename+'.csv.zip' , final_location+filename+'.csv.zip')

Here's the thing..

I'm able to open my csv from the zip after I create it. I'm unable to open the csv after I'm copying the file.

I currently ran out of ideas. tried copy2, didn't help. anyone have any idea what could go wrong?

thanks in advance!

Ofek.

Answer 1

You probably need to close zip_file before copying it; copyfile is going to open the file separately, and if the data isn't flushed to disk, it's going to copy an incomplete file (especially bad, since you'll likely omit the zip file metadata needed to interpret the contents).

The zipfile docs explicitly state:

You must call close() before exiting your program or essential records will not be written.

Since you're using Python 3, you can simplify further by making the close call implicit and guaranteed (even in the presence of exceptions), using context management (the with statement) to make a block that automatically closes the zip file when the block is exited.

Change your code to:

def make_done_file(user, filename):
    copyfile(network_location+filename+'.csv', test_location+filename+'.csv')
    with zipfile.ZipFile(test_location+filename+'.csv.zip', 'w') as zip_file:
        zip_file.write(test_location+filename+'.csv', compress_type=None, arcname=filename+'.csv')
    copyfile(test_location+filename+'.done', final_location+filename+'.done')
    copyfile(test_location+filename+'.csv.zip', final_location+filename+'.csv.zip')

Also, if you've got some other process looking for the .done file, you probably want to copy the zip first, then the .done , so the presence of the .done indicates a complete zip file has been copied, rather than "it's about to be copied, but if you read it too quickly, it might be incomplete".