most efficient way to temporarily store discontinuous data composing a larger file

Question

We are emulating a p2p network in java. So we divide the file into chunks (with checksums) so that the individual chunks can be recompiled into the original file once we have all the parts. What is the best way to store the individual parts while they are being downloaded?

I was thinking of just storing each chunk as a separate file...but if there are 20000 chunks, it would create as many files. is this the best way?

Thanks

Answer 1

Either keep chunks in memory or in files. There is no much to discuss here about. Findd the perfect ratio between chunks count and the actual size of it, to suit your needs.

Files sounds more reasonable as data would not be totally lost in case of application crash and continue of download would be possible.

Answer 2

I would write to memory until you reach some threshold, at which point you dump your memory to disk, and keep reading into memory. When the file transfer completes, you can take what is currently stored in memory, and concatenate it with what may have been stored on disk.