Sort huge file in java

Question

I've huge file with unique words in each line. Size of file is around 1.6 GB(I've to sort other files after this which are around 15GB). Till now, for smaller files I used Array.sort() . But for this file I get java.lang.OutOfMemoryError: Java heap space . I know the reason for this error. Is there any way instead of writing complete quick sort or merge sort program.

I read that Array.sort() uses Quicksort or Hybrid Sort internally. Is there any procedure like Array.sort() ??

If I have to write a program for sorting, which one should I use? Quicksort or Merge sort. I'm worried about worst case.

Answer 1

Depending on the structure of the data to store, you can do many different things.

In case of well structured data, where you need to sort by one or more specific fields (in which case system tools might not be helpful), you are probably better off using a datastore that allows sorting. MongoDB comes to mind as a good fit for this given that the size doesn't exceed few 100s of GBs. Other NoSQL datastores might also fit the bill nicely, although Mongo's simplicity of use and installation and support for JSON data makes it a really great candidate.

If you really want to go with the java approach, it gets real tricky. This is the kind of questions you ask at job interviews and I would never actually expect anybody to implement code. However, the general solution is merge sort (using random access files is a bad idea because it means insertion sort, ie, non optimal run time which can be bad given the size of your file).

By merge sort I mean reading one chunk of the file at a time small enough to fit it in memory (so it depends on how much RAM you have), sorting it and then writing it back to a new file on disk. After you read the whole file you can start merging the chunk files two at a time by reading just the head of each and writing (the smaller of the two records) back to a third file. Do that for the 'first generation' of files and then continue with the second one until you end up with one big sorted file. Note that this is basically a bottom up way of implementing merge sort, the academic recursive algorithm being the top down approach.

Note that having intermediate files can be avoided altogether by using a multiway merge algorithm . This is typically based on a heap/priority queue, so the implementation might get slightly more complex but it reduces the number of I/O operations required.

Please also see these links .

Implementing the above in java shouldn't be too difficult with some careful design although it can definitely get tricky. I still highly recommend an out-of-the-box solution like Mongo.

Answer 2

As it turns out, your problem is that your heap cannot accommodate such a large array, so you must forget any solution that implies loading the whole file content in an array (as long as you can't grow your heap).

So you're facing streaming. It's the only (and typical) solution when you have to handle input sources that are larger than your available memory. I would suggest streaming the file content to your program, which should perform the sorting by either outputting to a random access file (trickier) or to a database.

Answer 3

I'd take a different approach.

Given a file, say with a single element per line, I'd read the first n elements. I would repeat this m times, such that the amount of lines in the file is n * m + C with C being left-over lines.

When dealing with Integers , you may wish to use around 100,000 elements per read, with Strings I would use less, maybe around 1,000. It depends on the data type and memory needed per element.

From there, I would sort the n amount of elements and write them to a temporary file with a unique name.

Now, since you have all the files sorted, the smallest elements will be at the start. You can then just iterate over the files until you have processed all the elements, finding the smallest element and printing it to the new final output.

This approach will reduce the amount of RAM needed and instead rely on drive space and will allow you to handle sorting of any file size.

Answer 4

Build the array of record positions inside the file (kind of index), maybe it would fit into memory instead. You need a 8 byte java long per file record. Sort the array, loading records only for comparison and not retaining (use RandomAccessFile ). After sorting, write the new final file using index pointers to get the records in the needed order.

This will also work if the records are not all the same size.