Outputing a large file(>5GB) to a .zip archive in Java

Question

I am trying to take a large, tab delimited .txt file that is around 6GB and turn it into a .xml file using the JAXB api. This part works fine, but when I try to instead put that .xml into a .zip using ZipOutputStream, the .xml becomes corrupted when I try and look at it after(but it works for smaller files).

Is there another way I can do this or would it be better to do the the compression manually after the process runs? Below is some of the code I am using when trying to make the .zip file.

IFSFile source = null;
IFSFileOutputStream target = null;
ZipOutputStream targetZip = null;

String targetName = "C:/test.zip";

source = new IFSFile(as400, sourceName);
BufferedReader readBuffer = new BufferedReader(new IFSFileReader(source));

target = new IFSFileOutputStream(as400, targetName, IFSFileOutputStream.SHARE_NONE, false);
targetZip = new ZipOutputStream(target);

ZipEntry ze = new ZipEntry("test.xml");
targetZip.putNextEntry(ze);

//JAXB stuff omitted, seems to be working as no problems with smaller files
while ((strRead = readBuffer.readLine()) != null) {
    currentRecord = new stuff;
    marshaller.marshal(currentRecord, targetZip);
}

targetZip.closeEntry();
targetZip.close();
readBuffer.close();

When looking for size limitations all I could find was that it would be corrupt if the .zp ended up larger than 4GB, which I don't think it will. Am I doing something wrong or should I be using something other than ZipOutputStream for this?

Answer 1

Use GZIPOutputStream instead, it's part of java.util (so standard) and has better compression in (AFAIK) all cases, and will definitely handle files larger than 4Gb.

I think the ZipOutputStream might not be an implementation of ZIP64, if so it the original ZIP format has the 4Gb limit, as this is the maximum on a 32bit filesystem.

Answer 2

I think 4gb is the limit set by java did u try java 7? I read this may be bug in java also.Please also check truezip