Java：如何快速从大型文本文件中提取匹配行？

Question

Although aware that there are plenty of offered solutions to my problem in general , I am still not satisfied with the runtime they require in my special case.

Consider a 35G large text file in FASTA format, like this:

>Protein_1 So nice and cute little fella
MTTKKCLQKFHLESLGKLGDSFLKYAISIQLFKSYENHYEGLPSIKKNKIISNAALFKLG 
YARKILRFIRNEPFDLKVGLIPSDNSQAYNFGKEFLMPSVKMCSRVK*
>Protein_2 Fancy incredible description of its function
MADDSKFCFFLVSTFLLLAVVVNVTLAANYVPGDDILLNCGGPDNLPDADGRKWGTDIGS
[…] etc.

I need to extract the > lines only .

Using grep '>' proteins.fasta > protein_descriptions.txt to achieve this takes only a couple of minutes.

But using Java 7 this is now already running for over 90 minutes:

public static void main(String[] args) throws Exception {
    BufferedReader fastaIn = new BufferedReader(new FileReader(args[0]));
    List<String> l = new ArrayList<String>();
    String str;
    while ((str = fastaIn.readLine()) != null) {
        if (str.startsWith(">")) {
            l.append(str);
        }
    }
    fastaIn.close();
    // …
}

Does anyone have an idea of how to speed this up to grep performance?

Your help will be much appreciated. Cheers!

Answer 1

If you write it to the outfile immediatelly instead of accumulating objects in the memory it will improve performance (and will be more like what you did with grep anyway).

...
BufferedWriter fastaOut = new BufferedWriter(new FileWriter(args[1]));
...
while ((str = fastaIn.readLine()) != null) {
        if (str.startsWith(">")) {
            fastaOut.write(str);
            fastaOut.newLine();
        }
    }
...    
fastaOut.close();

Answer 2

The biojava.org provides a fasta reader. For reading huge files you would have to consider using a SeekableByteChannell and using the ByteBuffers. The biojava library uses bytebuffers.

Answer 3

You could probably speed this up considerably using multiple threads. If the file is X bytes long, and you have n threads, you start each thread at X/n intervals, and read X/n bytes. You will want to synchronize your ArrayList to ensure your results are added correctly