python making zip file of entire directories stored in list - individually

Question

I'm trying to automate some file collecting and zipping for my job. And I am having some trouble. I've tried using zipfile and shutil. Code below. Im having the same issue on with modules. Im feeding a list of directories, each with maybe 30mb max work of files. I want to zip each directory, with all its files, and save the zipped file in the directory thats being zipped.

eg:

+
|____dir_a
|        |__file_a
|        |__file_b
|        |__dir_a.zip
|
|____dir_b
         |___file_ba
         |___file_bb
         |___file_bc
         |___dir_b.zip

the problem, it seems, is that when i do the recommended steps in a for loop after gathering all the main directories into a list: [dir_a, dir_b] the zipping never seems to stop. It looks to me, like its trying to zip each file into one big file? or maybe just keep rezipping the same directory over and over? im not sure whats going on. But itll keep going until it eats up all the disk space.

here is the seemingly simple code im using:

def main(directory):

    if directory[-1] in ['/', '\\']:
        directory = directory[:-1]

    # Call the function to retrieve all files and folders of the assigned directory
    filePaths = [f.path for f in os.scandir(directory) if f.is_file()]

    # Set the zip file name
    zipFileName = os.path.split(directory)[-1] + ".zip"

    # printing the list of all files to be zipped
    print('The following list of files will be zipped:')
    for fileName in filePaths:
        print(fileName)

    # # writing files to a zipfile
    # with zipfile.ZipFile(directory + '/' + zipFileName, 'w', zipfile.ZIP_DEFLATED) as zip_file:
    #     # writing each file one by one
    #     for file in filePaths:
    #         zip_file.write(file)
    # print(directory+'.zip file is created successfully!')

    shutil.make_archive(directory + '/' + zipFileName[:-4], 'zip', directory)

    return directory + '/' + zipFileName

root = '/opt/folder_a/nested_folder/
for directory in [root + 'dir_a/', root + 'dir_b/']:
    main(directory)

so what im finding, is if there are more than like 5 directories, the zip files are mostly small. but then there will be the occasional zip file, that gets stuck... and ive watched it grow to around 6gb before i killed it. and in this directory with a 6gb zip file, the largest file size was only 230kb. Was mostly text files. whats happening here?

Answer 1

Rather than using shutil.make_archive you can use ZipFile (documentation here ) to stream your data into a zip file. I tried it out with several directories that have .zip files and .tar.gz and it worked just

In your case it would probably be

from zipfile import ZipFile

# create a ZipFile object
with ZipFile('sampleDir.zip', 'w') as zipObj:
   # Iterate over all the files in directory
   for folderName, subfolders, filenames in os.walk(dirName):
       for filename in filenames:
           #create complete filepath of file in directory
           filePath = os.path.join(folderName, filename)
           # Add file to zip
           zipObj.write(filePath)

Let me know if that works.