使用序列化二叉搜索树对海量数据进行排序

Question

I have 50 GB structured (as key/value) data like this which are stored in a text file (input.txt / keys and values are 63 bit unsigned integers);

3633223656935182015 2473242774832902432
8472954724347873710 8197031537762113360
2436941118228099529 7438724021973510085
3370171830426105971 6928935600176631582
3370171830426105971 5928936601176631564

I need to sort this data as keys in increasing order with the minimum value of that key. The result must be presented in another text file (data.out) under 30 minutes. For example the result must be like this for the sample above;

2436941118228099529 7438724021973510085
3370171830426105971 5928936601176631564
3633223656935182015 2473242774832902432
8472954724347873710 8197031537762113360

I decided that;

• I will create a BST tree with the keys from the input.txt with their minimum value, but this tree would be more than 50GB. I mean, I have time and memory limitation at this point.

• So I will use another text file (tree.txt) and I will serialize the BST tree into that file.

• After that, I will traverse the tree using in-order traverse and write the sequenced data into data.out file.

My problem is mostly with the serialization and deserialization part. How can I serialize this type of data? and I want to use the INSERT operation on the serialized data. Because my data is bigger than memory. I can't perform this in the memory. Actually I want to use text files as a memory.

By the way, I am very new to this kind of stuffs. If is there a conflict with my algorithm steps, please warn me. Any thought, technique and code samples would be helpful.

OS: Linux
Language: C
RAM: 6 GB

Note: I am not allowed to use built-in functions like sort and merge.

Answer 1

Considering, that your files seems to have the same line size around 40 chars giving me around 1250000000 lines in total, I'd split the input file into smaller, by a command:

split -l 2500000 biginput.txt

then I'd sort each of them

for f in x*; do sort -n $f > s$f; done

and finally I'd merge them by

sort -m sx* > bigoutput.txt