保留熊猫中的滋扰菌群

Question

I have a moderate sized data set I am processing with pandas. It has around 600,000 rows.

It has three "id" variables: "gene_id", "gene_name" and "transcript_id", and then a number of numerical columns which are determined at run-time.

In [129]: df.head().todict()
{u'utr3_count': {8: 2.0, 30: 1.0, 29: 2.0, 6: 2.0, 7: 2.0}, 
 u'gene_id': {8: u'ENSG00000188157', 30: u'ENSG00000160087', 29: u'ENSG00000176022', 6: u'ENSG00000188157', 7: u'ENSG00000188157'}, 
 u'utr3_enrichment': {8: 2.1449912126499999, 30: 1.14988290398, 29: 1.0484234234200001, 6: 2.1449912126499999, 7: 2.1449912126499999},
 u'transcript_id': {8: u'ENST00000379370', 30: u'ENST00000450390', 29: u'ENST00000379198', 6: u'ENST00000379370', 7: u'ENST00000379370'},
 u'expression': {8: 0.13387876534027521, 30: 0.514855687606112, 29: 0.79126387397064091, 6: 0.13387876534027521, 7: 0.13387876534027521}, 
 u'gene_name': {8: u'AGRN', 30: u'UBE2J2', 29: u'B3GALT6', 6: u'AGRN', 7: u'AGRN'}}

I want to get the mean of the replicates for each "transcript_id". But doing so by grouping on "transcript_id" means that I lose the information on "gene_id" and "gene_name" as they are classed as nuisance columns.

If I group on all three columns, I immediately get MemoryError, even on a big box (128GB), as pandas tries to do the calculation for every combination of the values in the three columns, even though this is definitely not necessary: each "transcript_id" maps to a single "gene_id" and a single "gene_name".

Is there a way to do the groupby just on transcript_id without losing the information in the other columns?

Answer 1

Simple Solution:

Store transcript_id , gene_id and gene_name in a separate DataFrame (say metadata ):

metadata = df[['transcript_id', 'gene_id', 'gene_name']].copy()
# .copy() is important!

groupby on transcript_id as you do now, and perform your calculations (say agg_df ). After they are done, merge the two frames together:

pd.merge(agg_df, metadata, how='left', on='transcript_id)

This works because

... each "transcript_id" maps to a single "gene_id" and a single "gene_name"

---

Alternate Solution:

Read the file (assuming you are reading from csv) in chunks using pd.read_csv(file_path, chunksize = <some integer, say 5e4>) . groupby on all three columns, (you won't run into MemoryError now because you are only reading part of the data) and keep running totals and running counts. Divide the totals by the counts at the end. Pseudo code:

totals = pd.DataFrame()
counts = pd.DataFrame()
df = pd.read_csv(file_path, chunksize=5e4)
for chunk in df:
    grouped = chunk.groupby(['transcript_id', 'gene_id', 'gene_name'])
    totals = totals.add(grouped.sum())
    counts = counts.add(grouped.count())
means = totals/counts

This will work as long as you need some measure that can be computed in bits and pieces, like sums, counts, products, cumulative sums and products. But anything like percentiles or

---

Another solution (slightly harder): Merge the columns transcript_id , gene_id and gene_name in another column, say merged_id and groupby on merged_id . Split the column up into the components at the end of your calculations.

---

Ps. I recommend using the Simple Solution.