sql 或 python 有什么快速的方法吗？

Question

I have a dataset of size 1TB containing 3 columns and about 20 billion rows. I would like to split this data in some random order into two sub datas in approximately 80/20 chunks. However, the two data should be non-overlapping meaning no entry in one chunk should appear in another chunk. An entry in one column of one chunk should not appear in any column of the other chunk.As an example, suppose an example data is:

fruit apple seeds
vegetable carrot yellow
crops fruit lettuce
green onion vegetable
lettuce red health

The two subdata can be

fruit apple seeds
crops fruit lettuce
lettuce red health

and

vegetable carrot yellow
green onion vegetable

Is there any efficient way to do this for such a large data?

Answer 1

You can just iterate over the file and randomly assign rows to sub-data-1 and sub-data-2 according to the proportions you've laid out.

import random
with open('large_file', 'r') as lf, 
open('s1', 'w') as s1, open('s2', 'w') as s2:
    for line in lf:
        if random.random() < 0.8:
            s1.write(line)
        else:
            s2.write(line)