Using LZ4 Compression in Java for multiple files

Question

I'm trying to compress multiple files into a single archive but with my current code, it only compresses it into a single blob inside the zip. Does anyone know how to segment the files with LZ4?

public void zipFile(File[] fileToZip, String outputFileName, boolean activeZip)
{
    try (FileOutputStream fos = new FileOutputStream(new File(outputFileName), true);
            LZ4FrameOutputStream lz4fos = new LZ4FrameOutputStream(fos);)
    {
        for (File a : fileToZip)
        {
            try (FileInputStream fis = new FileInputStream(a))
            {
                byte[] buf = new byte[bufferSizeZip];
                int length;
                while ((length = fis.read(buf)) > 0)
                {
                    lz4fos.write(buf, 0, length);                                             
                }
            }
        }
    }
    catch (Exception e)
    {
        LOG.error("Zipping file failed ", e);
    }
}

Answer 1

LZ4 algorithm is close with LZMA . In case you can use LZMA then you can create zip archive with LZMA compression.

List<Path> files = Collections.emptyList();
Path zip = Paths.get("lzma.zip");

ZipEntrySettings entrySettings = ZipEntrySettings.builder()
                                                 .compression(Compression.LZMA, CompressionLevel.NORMAL)
                                                 .lzmaEosMarker(true).build();
ZipSettings settings = ZipSettings.builder().entrySettingsProvider(fileName -> entrySettings).build();

ZipIt.zip(zip)
     .settings(settings)
     .add(files);

See details in zip4jvm

Answer 2

LZ4 compresses a stream of bytes. You would need to archive your multiple files into a single archive such as a Tar Archive, then feed it into the LZ4 compressor.

I created a Java library that does this for you https://github.com/spoorn/tar-lz4-java .

If you want to implement it yourself, here's a technical doc that includes details on how to LZ4 compress a directory using TarArchive from Apache Commons and lz4-java:https://github.com/spoorn/tar-lz4-java/blob/main/SUMMARY.md#lz4