Creating a directory containing large files in Java

Question

I have a problem statement, wherein I need to compress all contents of a directory (that might contain other subdirectories, and files and folders) into zip files of given size . I have been able to accomplish this, but there is a single constraint I am not able to wrap my head around, and that is:

If the input file size (lets call this file F ) is bigger than the size of the given zip file, how do I compress that? I thought of multiple ways, but have doubts:

1. Can I break the large file F into smaller parts, and compress each of those parts across multiple zip files?

Here is a piece of code I am currently working on:

    public static void main(String[] args) throws IOException {
    System.out.println("HelloWorld");
    String sourceFile = "zipTest";
    FileOutputStream fos = new FileOutputStream("dirCompressed.zip");
    ZipOutputStream zipOut = new ZipOutputStream(fos);
    File fileToZip = new File(sourceFile);

    zipFile(fileToZip, fileToZip.getName(), zipOut);
    zipOut.close();
    fos.close();
}

private static void zipFile(File fileToZip, String fileName, ZipOutputStream zipOut) throws IOException {
    if (fileToZip.isHidden()) {
        return;
    }
    if (fileToZip.isDirectory()) {
        if (fileName.endsWith("/")) {
            zipOut.putNextEntry(new ZipEntry(fileName));
            zipOut.closeEntry();
        } else {
            zipOut.putNextEntry(new ZipEntry(fileName + "/"));
            zipOut.closeEntry();
        }
        File[] children = fileToZip.listFiles();
        for (File childFile : children) {
            zipFile(childFile, fileName + "/" + childFile.getName(), zipOut);
        }
        return;
    }
    FileInputStream fis = new FileInputStream(fileToZip);
    ZipEntry zipEntry = new ZipEntry(fileName);
    zipOut.putNextEntry(zipEntry);
    byte[] bytes = new byte[1024];
    int length;
    while ((length = fis.read(bytes)) >= 0) {
        zipOut.write(bytes, 0, length);
    }
    fis.close();
}

Answer 1

APPROACH #0 - USING JDK ONLY After searching the net, i came across the following post from oracle jdk site - https://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/zipfilesystemprovider.html

where they talk about treating zip file(s) as a file system, and by doing so, they have provided the key to compressing files and folders recursively as you see in the example mentioned in the link, would be treated as a File system which can be accessed using java.nio (which even I was unaware of.!).

Furthermore, you can just use File.copy to copy/move any file in and out of file system as you would do normally, but, here programmatically.

NOTE: if time permits, i shall create a sample program making us of the same and post it here, until then, you can definitely take a look at this approach.

Approach #1 - NOT USED BECAUSE IT MAKES USE OF EXTERNAL LIBRARY

Please take a look at the following repo below, where, they have many cases handled, if you are OK in making use of external lib in your project(as you have not mentioned in the question that you are confined to make use of vanilla java only), then, this should help you out. zip4j - this is available here https://github.com/srikanth-lingala/zip4j

they have also taken the example of 10MB as mentioned by you in your question.

zipFile.createSplitZipFile(filesToAdd, zipParameters, true, 10485760); // using 10MB in this example

you can also get it from maven

<dependency>
    <groupId>net.lingala.zip4j</groupId>
    <artifactId>zip4j</artifactId>
    <version>2.2.4</version>
</dependency>

<Adding 6 mandatory characters because accidentally edited your code to write the example of using zip4j>



Here, is an example and the output, 1.dat is 109mb file 
-rw-r--r--  1 apachemain  staff   109M Nov 15 18:00 1.dat

package org.example;

import net.lingala.zip4j.ZipFile;
import net.lingala.zip4j.exception.ZipException;
import net.lingala.zip4j.model.ZipParameters;
import net.lingala.zip4j.model.enums.CompressionMethod;
import net.lingala.zip4j.model.enums.EncryptionMethod;

import java.io.File;
import java.util.Arrays;
import java.util.List;

public class CreateSplitZipFile {
    public CreateSplitZipFile() {

        try {
            // Initiate ZipFile object with the path/name of the zip file.

            // Build the list of files to be added in the array list
            // Objects of type File have to be added to the ArrayList
            ZipParameters zipParameters = new ZipParameters();
            zipParameters.setEncryptFiles(true);
            zipParameters.setEncryptionMethod(EncryptionMethod.AES);
            zipParameters.setCompressionMethod(CompressionMethod.STORE);

            List<File> filesToAdd = Arrays.asList(
                    new File("1.dat")
            );

            ZipFile zipFile = new ZipFile("filename.zip", "password".toCharArray());
            zipFile.createSplitZipFile(filesToAdd, zipParameters, true, 10485760); // using 10MB in this example


        } catch (ZipException e) {
            e.printStackTrace();
        }
    }

    /**
     * @param args
     */
    public static void main(String[] args) {
        new CreateSplitZipFile();
    }


}

OUTPUT
=======
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z01
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z02
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z03
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z04
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z05
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z06
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z07
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z08
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z09
-rw-r--r--  1 apachemain  staff    10M Nov 16 10:26 filename.z10
-rw-r--r--  1 apachemain  staff   8.6M Nov 16 10:26 filename.zip


am also seeing the following exception after the creation of the zip files.

net.lingala.zip4j.exception.ZipException: java.io.FileNotFoundException: null/filename.z01 (No such file or directory)
    at net.lingala.zip4j.tasks.AsyncZipTask.performTaskWithErrorHandling(AsyncZipTask.java:49)
    at net.lingala.zip4j.tasks.AsyncZipTask.execute(AsyncZipTask.java:36)
    at net.lingala.zip4j.ZipFile.createSplitZipFile(ZipFile.java:152)
    at org.example.CreateSplitZipFile.<init>(CreateSplitZipFile.java:32)
    at org.example.CreateSplitZipFile.main(CreateSplitZipFile.java:44)
Caused by: java.io.FileNotFoundException: null/filename.z01 (No such file or directory)