带有数据框的熊猫重新索引

Question

I have a DataFrame with a multiindex with three levels, for instance:

                   COL1  COL2  ...
CHROM  POS  LABEL                 
chr1   43   strA   ...   ...   ...
            strB   ...   ...   ...
       66   strC   ...   ...   ...
            strB   ...   ...   ...
chr2   29   strD   ...   ...   ...
...    ...  ...    ...   ...   ...

and a Series with a multiindex with the first two levels of the DataFrame index:

            VAL
CHROM  POS     
chr1   43   v1
       66   v2
chr2   29   v3
...    ...  ...

I would like to add a column with the Series to the DataFrame, repeating the values v1, v2... for every index whose first two levels match, like this:

                   COL1  COL2  NEW  ...
CHROM  POS  LABEL                 
chr1   43   strA   ...   ...   v1   ...
            strB   ...   ...   v1   ...
       66   strC   ...   ...   v2   ...
            strB   ...   ...   v2   ...
chr2   29   strD   ...   ...   v3   ...
...    ...  ...    ...   ...   ...  ...

Note that the Series has no missing rows, that is, all (CHROM,POS) in the DataFrame are also in the Series. I have a working solution:

pandas.Series(variant_db.index.map(lambda i: cov_per_sample[sample].loc[i[:2]]), index=variant_db.index)

but, because of that lambda, it is quite slow for big data (hundreds of thousands of rows). I tried with the much faster:

df['NEW'] = s.reindex(df.index, method='ffill')

but in this way there are many NaNs in df['NEW'], which should not happen. Using method='bfill' I get NaNs in different positions, but some rows get NaNs in both cases, so even using both does not work.

I would like a way to do this using library function only, for efficiency. Can anyone help?

Answer 1

You can try this very simple solution with your big data for performance:

df1=pandas.DataFrame([
{'CHROM':'chr1','POS':43,'LABEL':'strA'},
{'CHROM':'chr1','POS':43,'LABEL':'strB'},
{'CHROM':'chr1','POS':66,'LABEL':'strC'},
{'CHROM':'chr1','POS':66,'LABEL':'strB'},
{'CHROM':'chr2','POS':29,'LABEL':'strD'}])

df2=pandas.DataFrame([
{'CHROM':'chr1','POS':43,'VAL':'v1'},
{'CHROM':'chr1','POS':66,'VAL':'v2'},
{'CHROM':'chr2','POS':29,'VAL':'v3'}])

for i,r in df2.iterrows():
    df1.ix[(df1['CHROM']==r['CHROM']) & (df1['POS']==r['POS']),'NEW']=r['VAL']

Or using indexes:

df1=pandas.DataFrame([
{'CHROM':'chr1','POS':43,'LABEL':'strA','COL':''},
{'CHROM':'chr1','POS':43,'LABEL':'strB','COL':''},
{'CHROM':'chr1','POS':66,'LABEL':'strC','COL':''},
{'CHROM':'chr1','POS':66,'LABEL':'strB','COL':''},
{'CHROM':'chr2','POS':29,'LABEL':'strD','COL':''}]).set_index(['CHROM','POS','LABEL'])

df2=pandas.DataFrame([
{'CHROM':'chr1','POS':43,'VAL':'v1'},
{'CHROM':'chr1','POS':66,'VAL':'v2'},
{'CHROM':'chr2','POS':29,'VAL':'v3'}]).set_index(['CHROM','POS'])

for i,r in df2.iterrows():
    df1.ix[(i[0],i[1]),'NEW']=r['VAL']

Answer 2

this is what pandas is all about. use the indices to your advantage.

df1 = df1.reset_index().set_index(['CHROM', 'POS'])
df1['NEW'] = df2.VAL

Answer 3

Elaborating on the answer provided by @acushner, something like this should work

midx = pd.MultiIndex.from_product(
    [["chr1","chr2"],[43,66,29],["strA","strB","strC"]],
    names=["CHROM", "POS", "LABEL"]
    )

df = pd.DataFrame(random.random([18,2]), index=midx)

midx2 = pd.MultiIndex.from_product([["chr1","chr2"],[43,66,29]],
                                   names=["CHROM", "POS"])
ser = pd.Series(random.random(6), index=midx2)

df = df.reset_index().set_index(['CHROM', 'POS'])
df[2] = ser
df.set_index("LABEL", append=True, inplace=True)