take a subset of a pandas data frame based on the column names

Question

I have a pd.DataFrame with 140 samples (columns) and ~27000 SNPs (rows). The column names each have a population name plus a number (eg 'FLFL04' or 'MI03' ) with 6 different populations and differing numbers of samples in populations.

I would like to take subsets of the respective populations based on the population names for further calculations (Hardy-Weinberg exact test); I could do it with a loop and regex, but was hoping that there is a quicker solution for this. Is there a way to create subsets based on column names (as opposed to their content)?

EDIT: my current approach is the following:

(any pd.DataFrame will do, with the following columns:

data.columns = ['FLFL04', 'FLFL08', 'FLFL08replicate', 'FLFL10', 'FLFL13', 'FLFL14', 'FLFL15', 'FLFL15replicate', 'FLFL16', 'FLFL17', 'FLFL17replicate', 'FLFL19', 'FLFL20', 'FLFL20replicate', 'FLFL21', 'FLFL23', 'FLFL26', 'FLFL28', 'FLFL28replicate', 'FLFL29', 'FLFL29replicate', 'FLFL30', 'HSPQ01', 'HSPQ01replicate', 'HSPQ01replicate2', 'HSPQ02', 'HSPQ02replicate', 'HSPQ02replicate2', 'HSPQ03', 'HSPQ04', 'HSPQ04replicate', 'HSPQ04replicate2', 'HSPQ06', 'HSPQ07', 'HSPQ08', 'HSPQ09', 'HSPQ09replicate', 'HSPQ10', 'HSPQ10replicate', 'HSPQ11', 'HSPQ12', 'HSPQ13', 'HSPQ14', 'HSPQ15', 'HSPQ16', 'HSPQ17', 'HSPQ18', 'HSPQ19', 'HSPQ21', 'HSPQ22', 'HSPQ22replicate', 'KFO1', 'KFO2', 'KFO3', 'KFO4', 'KFO5', 'KFO8', 'MI01', 'MI02', 'MI03', 'MI03replicate', 'MI03replicate2', 'MI04', 'MI05', 'MI06', 'MI06replicate', 'MI06replicate2', 'MI08', 'MI09', 'MI09replicate', 'MI09replicate2', 'MI10', 'MI11', 'MI12', 'MI12replicate', 'MI13', 'MI13replicate', 'MI14', 'MI15', 'MI16', 'MI16replicate', 'MI17', 'MI18', 'MI19', 'MI20', 'MI21', 'SFQ01', 'SFQ02', 'SFQ03', 'SFQ03replicate', 'SFQ05', 'SFQ05replicate', 'SFQ06', 'SFQ06replicate', 'SFQ07', 'SFQ08', 'SFQ08replicate', 'SFQ09', 'SFQ09replicate', 'SFQ10', 'SFQ10replicate', 'SFQ11', 'SFQ13', 'SFQ14', 'SFQ15', 'SFQ16', 'SFQ17', 'SFQ21', 'SFQ23', 'SFQ24', 'SFQ25', 'SFQ26', 'WWA01', 'WWA01replicate', 'WWA01replicate2', 'WWA03', 'WWA03replicate', 'WWA03replicate2', 'WWA04', 'WWA05', 'WWA05replicate', 'WWA05replicate2', 'WWA07', 'WWA08', 'WWA08replicate', 'WWA09', 'WWA10', 'WWA12', 'WWA17', 'WWA17replicate', 'WWA18', 'WWA21', 'WWA23', 'WWA24', 'WWA25', 'WWA25replicate', 'WWA26', 'WWA27', 'WWA28', 'WWA30']

def get_pop_subset(pop_list, pop_name): 
    pop_result_list = []
    for i, pop in enumerate(data.columns):
    curr_pop = re.findall('([A-Z]+)', pop)[0]
    if curr_pop == pop_name:
        pop_result_list.append(pop)
    return pop_result_list

pops = ['FLFL', 'HSPQ', 'KFO', 'MI', 'SFQ', 'WWA']
subsets = []
for val in pops:
    subsets.append(get_pop_subset(data.columns, val))

for val in subsets:
    print data[val]

I then call other funcs instead of

    print data[val]

and append each to a new df. While this works, I was hoping to get a quicker and probably more efficient solution

thanks, martin

Answer 1

Couldn't you achieve the same thing using the built-in DataFrame method "filter" with the argument "regex"? For example,

df2 = df.filter(regex='FLFL')

returns a new DataFrame with all of the columns starting with FLFL.

Answer 2

Okay, for your case, I'd use groupby . You can pass a function to it with axis=1 to loop over the columns (see here in the docs):

>>> df 
   FLFL04  FLFL29rep  HSPQ12  MI03repl  MI16repl  SFQ10re  WWA05r
0       0          3       6         9        12       15      18
1       1          4       7        10        13       16      19
2       2          5       8        11        14       17      20
>>> df.groupby(lambda x: re.match("[A-Z]+", x).group(), axis=1)
<pandas.core.groupby.DataFrameGroupBy object at 0x9ae660c>
>>> grouped = df.groupby(lambda x: re.match("[A-Z]+", x).group(), axis=1)

And then we can loop over the groups:

>>> for name, group in grouped:
    print 'group name:', name
    print 'dataframe:'
    print group
...     
group name: FLFL
dataframe:
   FLFL04  FLFL29rep
0       0          3
1       1          4
2       2          5
group name: HSPQ
dataframe:
   HSPQ12
0       6
1       7
2       8
group name: MI
dataframe:
   MI03repl  MI16repl
0         9        12
1        10        13
2        11        14
group name: SFQ
dataframe:
   SFQ10re
0       15
1       16
2       17
group name: WWA
dataframe:
   WWA05r
0      18
1      19
2      20

Or turn it into a dictionary:

>>> pprint.pprint(dict(list(grouped)))
{'FLFL':    FLFL04  FLFL29rep
0       0          3
1       1          4
2       2          5,
 'HSPQ':    HSPQ12
0       6
1       7
2       8,
 'MI':    MI03repl  MI16repl
0         9        12
1        10        13
2        11        14,
 'SFQ':    SFQ10re
0       15
1       16
2       17,
 'WWA':    WWA05r
0      18
1      19
2      20}