如何有效地写入大量文件

Question

I am trying to write a program to split a large collection of gene sequences into many files based on values inside a certain segment of each sequence. For example the sequences might look like

AGCATGAGAG...
GATCAGGTAA...
GATGCGATAG...
... 100 million more

The goal is then to split the reads into individual files based on the sequences from position 2 to 7 (6 bases). So we get something like

AAAAAA.txt.gz
AAAAAC.txt.gz
AAAAAG.txt.gz
...4000 more

Now naively I have implemented a C++ program that

• reads in each sequence
• opens the relevant file
• writes in the sequence
• closes the file

Something like

#include <zlib.h>

void main() {
    SeqFile seq_file("input.txt.gz");
    string read;

    while (read = seq_file.get_read) {
        string tag = read.substr(1, 6);
        output_path = tag + "txt.gx";

        gzFile output = gzopen(output_path.c_str(), "wa");
        gzprintf(output, "%s", read);
        gzclose(output);
    }
}

This is unbearably slow compared to just writing the whole contents into a single other file.

What is the bottleneck is this situation and how might I improve performance given that I can't keep all the files open simultaneously due to system limits?

Answer 1

Since opening a file is slow, you need to reduce the number of files you open. One way to accomplish this is to make multiple passes over your input. Open a subset of your output files, make a pass over the input and only write data to those files. When you're done, close all those files, reset the input, open a new subset, and repeat.

Answer 2

The bottleneck is opening and closing of the output file. If you can move this out of the loop somehow, eg by keeping multiple output files open simultaneously, you program should speed up significantly. In the best case scenario it is possible to keep all 4096 files open at the same time but if you hit some system barrier even keeping a smaller number of files open and doing multiple passes through the file should be faster that opening and closing files in the tight loop.

Answer 3

The compressing might be slowing the writing down, writing to text files then compressing could be worth a try.

Opening the file is a bottleneck. Some of the data could be stored in a container and when it reaches a certain size write the largest set to the corresponding file.

Answer 4

I can't actually answer the question - because to do that, I would need to have access to YOUR system (or a reasonably precise replica). The type of disk and how it is connected, how much and type of memory and model/number of CPU will matter.

However, there are a few different things to consider, and that may well help (or at least tell you that "you can't do better than this").

First find out what takes up the time: CPU or disk-I/O?

Use top or system monitor or some such to measure what CPU-usage your application uses.

Write a simple program that writes a single value (zero?) to a file, without zipping it, for a similar size to what you get in your files. Compare this to the time it takes to write your gzip-file. If the time is about the same, then you are I/O-bound, and it probably doesn't matter much what you do.

If you have lots of CPU-usage, you may want to split the writing work into multiple threads - you obviously can't really do that with the reading, as it has to be sequential (reading gzip in mutliple threads is not easy, if at all possible, so let's not try that). Using one thread per CPU-core, so if you have 4 cores, use 1 to read, and three to write. You may not get 4 times the performance, but you should get a good improvement.

Quite certainly, at some point, you will be bound by the speed of the disk. Then the only option is to buy a better disk (if you haven't already got that!)