在第一列的R中查询内存不足的60gb tsv，哪个数据库/方法？

Question

I have 6 large tsv's matrices of 60gb (uncompressed) containing 20million rows x 501 columns: the first index/integer column that is basically the row number (so not even necessary), 500 columns are numerical (float, 4 decimals eg 1.0301). All tsv's have the same number of rows that correspond to each other.

I need to extract rows on rownumber.

I need is to extract up to 5,000 contiguous rows or up to 500 non-contiguous rows;so not millions. Hopefully, also have some kind of compression to reduce the size of 60gb so maybe no SQL? What would be the best way to do this?

• One method I tried is to seperate them into 100 gzipped files, index them using tabix, and then query them, but this is too slow for my need (500 random rows took 90 seconds).

• I read about the ff package, but have not found how to index by the first column?

• Are there other ways ?

Thanks so much.

Answer 1

I will use fread() from data.table package

Using the parameters skip and nrows you can play with the starting line to read ( skip ) or the number of rows to read ( nrows )

If you want to explore the tidyverse approach I recommend you this solution R: Read in random rows from file using fread or equivalent?