用于大型制表符/逗号分隔文本文件的python库

Question

I have some big genomic data files to analyze, which come in two forms, one individual dosage file like this:

id                      snp1    snp2    snp3    snp4    snp5    snp6
RS1->1000001    DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1->1000002    DOSE    1.291   1.998   1.998   1.998   1.830   1.335
RS1->100001     DOSE    1.992   1.998   1.998   1.998   1.830   1.335
RS1->100002     DOSE    1.394   1.998   1.998   1.998   1.830   1.335
RS1->10001      DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1->1001001    DOSE    1.904   1.998   1.998   1.998   1.830   1.335
RS1->1002001    DOSE    1.094   1.998   1.998   1.998   1.830   1.335
RS1->1003001    DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1->1004001    DOSE    1.994   1.998   1.998   1.998   1.830   1.335
RS1->1005002    DOSE    1.994   1.998   1.998   1.998   1.830   1.335

The other contains some summary info:

SNP         Al1 Al2 Freq1   MAF     Quality Rsq 
22_16050607 G   A   0.99699 0.00301 0.99699 0.00000
22_16050650 C   T   0.99900 0.00100 0.99900 0.00000
22_16051065 G   A   0.99900 0.00100 0.99900 0.00000
22_16051134 A   G   0.99900 0.00100 0.99900 0.00000
rs62224609  T   C   0.91483 0.08517 0.91483 -0.00000
rs62224610  G   C   0.66733 0.33267 0.66733 0.00000
22_16051477 C   A   0.99399 0.00601 0.99399 -0.00000
22_16051493 G   A   0.99900 0.00100 0.99900 -0.00000
22_16051497 A   G   0.64529 0.35471 0.64529 0.00000

The SNP column in the second file corresponds the snp1, snp2... in the first file. I need to use the summary info in the second file to do some quality check and selection, then apply some statistical analysis on the data in the first file accordingly.

The question is, is there a python library suitable for this task? Performance is vital here, because these are really huge files. Thanks!

Answer 1

For dealing with large files and data with high performance and efficient manipulation, there is really no better module than pandas

The following code will read your file into a DataFrame and allow easy manipulation:

import pandas as pd
data = 'my_data.csv'
df = pd.read_csv(data)

now df is an efficient dataframe containing your data! Also, you don't even need to say it's tab delimiter because pandas "sniffs" for the delimiter

Answer 2

There is the csv module. It's written with a C backend, so it should perform pretty well. That said, str.split might be even faster if the format is simple enough.

It seems to me that rather than using a CSV file to store the data, some sort of database is probably an even better bet.