在 Pandas 中创建新数据框的正确索引

Question

Basically I'm trying to rearrange a horrible csv file into usable information and I think I'm trying to cheat the slicing process which is resulting in a lot of indexing vs copy warnings and eventually the wrong result.

I have data that looks like this:

lipid1 #some of the names of lipids have commas in them which is an added challenge
tissue1,1
tissue2,6
tissue3,3
tissue4,2
tissue5,5


lipid2
tissue1,24
tissue2,15
tissue3,12
tissue4,14
tissue5,10

and I want to get it to be something like

        tissue1  tissue2  tissue3  tissue4  tissue5
lipid1  1        6        3        2        5
lipid2  24       15       12       14       10

Pretty sure this has an easy solution that I am overlooking because so far I've been using something like:

alldata = pd.DataFrame()
for file in glob.glob("All5tissuesPos.csv"):
    filename = file[:-4]
    tissue = file[:-7]

    dirty = pd.read_csv(filename+'.csv', sep='\n', header=None, names=['Arb'])
    #data = dirty['Arb'].str.split(',',expand=True)

    lipid = dirty.iloc[::6]['Arb'].copy()
    #lipid = dirty.iloc[lambda x:x.index%6 == 0]['Arb'].copy()

    data = dirty['Arb'].str.split(',',expand=True)

    t=data[data.index %6 != 0]

    tissue1 = t[t[0]== 'Tissue 1']
    tissue1 ['lipid'] = lipid
    alldata.append(tissue1)
    tissue1.to_csv('test.csv')

tissue1 at the last step does look like what I want, but since it's really just parts of another dataframe instead of a separate one (I think anyway) I get the warnings and when I go to append it nothing happens. What is this kind of code supposed to look like? Is there a faster way to do this for all 5 tissues at once?

Answer 1

You can simplify this a bunch. We'll use a trick of creating another column of the lipid, forward filling the value and then dropping the original row, which is no longer necessary. We then get to your dataset with a simple pivot. In my sample data I have a lipid with a messy name, including commas.

Here I use every 6 rows like your condition, but if the data are messier and some rows are missing you could just as easily use a condition with something like .str.contains('lipid') .

dirty = pd.read_csv('test.csv', sep='\n', header=None, names=['Arb'])

# Broadcast lipid name, drop that "header" row
dirty['lipid_name'] = dirty['Arb'].where(dirty.index%6 == 0).ffill()
dirty = dirty[dirty.index%6 != 0]

# Now we can split data properly
dirty = dirty.set_index('lipid_name')['Arb'].str.split(',', expand=True)

dirty.pivot(columns=0, values=1).rename_axis(None, axis=1)

---

                           tissue1 tissue2 tissue3 tissue4 tissue5
lipid_name                                                             
lipid11231,12312313,123123       1       6       3       2       5
lipid2                          24      15      12      14      10

---

Sample Data: test.csv

lipid11231,12312313,123123
tissue1,1
tissue2,6
tissue3,3
tissue4,2
tissue5,5
lipid2
tissue1,24
tissue2,15
tissue3,12
tissue4,14
tissue5,10