I'm trying to download a gzipped file (~390Mo) from a FTP file using java. But the program stops after reading a few lines.
Here is the minimal program for this problem:
import java.net.*;
import java.io.*;
import java.util.zip.GZIPInputStream;
public class Test
{
public static void main(String args[]) throws Exception
{
int count=0;
URL url=new URL("ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20100804/ALL.2of4intersection.20100804.sites.vcf.gz");
String line;
BufferedReader in= new BufferedReader(new InputStreamReader(new GZIPInputStream(url.openStream())));
while((line=in.readLine())!=null)
{
++count;
System.err.println("["+count+"] "+line);
}
in.close();
System.out.println("Done. nLines="+count);
}
}
Compile and run:
javac Test.java
java -Dftp.proxyHost=${MYPROXYHOST} -Dftp.proxyPort=${MYPROXYPORT} Test
And the output stops prematurely after the 1012th line:
(...)
[999] 1 750138 rs61770171 G A . PASS DP=2189;AF=0.083;CB=UM,BI;EUR_R2=0.129;AFR_R2=0.164
[1000] 1 750153 . T C . PASS DP=2555;AF=0.016;CB=UM,BI,BC;EUR_R2=0.167;AFR_R2=0.281
[1001] 1 750190 . C T . PASS DP=3515;AF=0.003;CB=UM,BI;EUR_R2=0.581;AFR_R2=0.575
[1002] 1 750235 . G A . PASS DP=3914;AF=0.019;CB=UM,BI,BC;EUR_R2=0.719;AFR_R2=0.733
[1003] 1 750436 . C T . PASS DP=598;AF=0.020;CB=BI,BC;EUR_R2=0.144;AFR_R2=0.355
[1004] 1 750511 . G A . PASS DP=806;AF=0.010;CB=BI,BC;AFR_R2=0.352
[1005] 1 750718 . G A . PASS DP=2751;AF=0.003;CB=UM,BI,BC;EUR_R2=0.54;AFR_R2=0.545
[1006] 1 750897 . G A . PASS DP=744;AF=0.010;CB=BI,BC;AFR_R2=0.479
[1007] 1 750946 . A G . PASS DP=873;AF=0.010;CB=BI,BC;AFR_R2=0.414
[1008] 1 751043 . G A . PASS DP=1522;AF=0.000;CB=BI,BC;EUR_R2=0.273
[1009] 1 751281 . T C . PASS DP=403;AF=0.010;CB=BI,BC;AFR_R2=0.178
[1010] 1 751343 . T A . PASS DP=1912;AF=0.117;CB=UM,BI;EUR_R2=0.683;AFR_R2=0.582
[1011] 1 751456 . T C . PASS DP=1775;AF=0.008;CB=UM,BI;EUR_R2=0.515;AFR_R2=0.332
[1012] 1
Done. nLines=1012
Why ? what's happening ?
Thanks for your help.
Pierre
Edit : I also tried using an InputStream rather than a Reader. It doesn't work:
import java.net.*;
import java.io.*;
import java.util.zip.GZIPInputStream;
public class Test
{
public static void main(String args[]) throws Exception
{
URL url=new URL("ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20100804/ALL.2of4intersection.20100804.sites.vcf.gz");
String line;
byte array[]=new byte[10];
int nRead=0;
InputStream in= new GZIPInputStream(url.openStream());
while((nRead=in.read(array))!=-1)
{
System.out.write(array,0,nRead);
}
in.close();
System.out.println("Done.");
}
}
ftp.1000genomes.ebi.ac.uk uses a special form of gzip compression that is not handled by GZIPInputStream ( See http://biostar.stackexchange.com/questions/6112/i-cant-download-a-file-from-the-1k-genomes-ftp-site/6114#6114 )
Using net.sf.samtools.util.BlockCompressedInputStream instead of GZipInputStream fixes the problem:
import java.net.*;
import java.io.*;
import java.util.zip.GZIPInputStream;
import net.sf.samtools.util.BlockCompressedInputStream;
public class Test
{
public static void main(String args[]) throws Exception
{
URL url=new URL("ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20100804/ALL.2of4intersection.20100804.sites.vcf.gz");
String line;
int nRead=0;
BufferedReader in= new BufferedReader(new InputStreamReader(new BlockCompressedInputStream(url.openStream())));
while((line=in.readLine())!=null)
{
System.out.println(line);
}
in.close();
System.out.println("Done.");
}
}
Do not use readline, but read the file into a byte[]! Else you get a lot of nasty String conversion errors!
byte[] buf = new byte[4096];
int bytesRead;
while( (bytesRead = in.read(buf)) >= 0 ) {
outFile.write(buf,0,bytesRead);
}
OK, this may help : the length of your output above (till line 1012) is exactly 65 536 bytes long. Strange coincidence isn't it ? Try this code on the .vcf file :
FileInputStream in = new FileInputStream("ALL.2of4intersection.20100804.sites.vcf");
in.skip(65534);
for (int i=0; i<10; i++) {
System.out.println("byte [" + (65534 + i) + "] = " + in.read());
}
I get the following output :
byte [65534] = 49
byte [65535] = 9
byte [65536] = -1
byte [65537] = -1
byte [65538] = -1
byte [65539] = -1
byte [65540] = -1
byte [65541] = -1
byte [65542] = -1
byte [65543] = -1
Also if you try the following command :
head -2070 ALL.2of4intersection.20100804.sites.vcf >test.vcf
you only get the first 65 536 bytes
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.