使用pandas的從長到長的數據集

Question

有很多問題有類似的標題，但我無法解決我對我的數據集的問題。

數據集：

ID   Country Type Region Gender IA01_Raw  IA01_Class1  IA01_Class2 IA02_Raw IA02_Class1 IA02_Class2 QA_Include QA_Comments

SC1  France  A    Europe Male   4         8            1            J         4            1           yes       N/A
SC2  France  A    Europe Female 2         7            2            Q         6            4           yes       N/A
SC3  France  B    Europe Male   3         7            2            K         8            2           yes       N/A
SC4  France  A    Europe Male   4         8            2            A         2            1           yes       N/A
SC5  France  B    Europe Male   1         7            1            F         1            3           yes       N/A
ID6  France  A    Europe Male   2         8            1            R         3            7           yes       N/A
ID7  France  B    Europe Male   2         8            1            Q         4            6           yes       N/A
UC8  France  B    Europe Male   4         8            2            P         4            2           yes       N/A

所需輸出：

ID   Country Type Region Gender IA Raw Class1 Class2 QA_Include QA_Comments

SC1  France  A    Europe Male   01 K   8      1      yes        N/A
SC1  France  A    Europe Male   01 L   8      1      yes       N/A
SC1  France  A    Europe Male   01 P   8      1      yes       N/A
SC1  France  A    Europe Male   02 Q   8      1      yes       N/A
SC1  France  A    Europe Male   02 R   8      1      yes       N/A
SC1  France  A    Europe Male   02 T   8      1      yes       N/A
SC1  France  A    Europe Male   03 G   8      1      yes       N/A
SC1  France  A    Europe Male   03 R   8      1      yes       N/A
SC1  France  A    Europe Male   03 G   8      1      yes       N/A
SC1  France  A    Europe Male   04 K   8      1      yes       N/A
SC1  France  A    Europe Male   04 A   8      1      yes       N/A
SC1  France  A    Europe Male   04 P   8      1      yes       N/A
SC1  France  A    Europe Male   05 R   8      1      yes       N/A
....

---

在數據集中我列的名稱為IA [X] _NAME ，其中X = 1..9 ， NAME = Raw，Class1和Class2 。

我想要做的是只是轉換這些列，使它看起來像必需輸出中顯示的表，即IA將顯示X值，就像這樣原始 ， 類將顯示他們的透視值。

所以為了實現它，我將列切成：

idVars = list(excel_df_final.columns[0:40]) + list(excel_df_final.columns[472:527]) #These contain columns like ID, Country, Type etc
valueVars = excel_df_final.columns[41:472].tolist() #All the IA_ columns

我不知道這一步是否必要，但是這給了我完美的切片，但是當我把它放入melt它不能正常工作。 我已經嘗試了幾乎所有其他問題中可用的方法。

pd.melt(excel_df_final, id_vars=idVars,value_vars=valueVars)

我也試過這個：

excel_df_final.set_index(idVars)[41:472].unstack()

但沒有工作，這里是廣泛的長期實施，也沒有工作：

pd.wide_to_long(excel_df_final, stubnames = ['IA', 'Raw', 'Class1', 'Class2'], i=idVars, j=valueVars)

我得到的錯誤是：

ValueError：操作數無法與形狀一起廣播（95，）（431，）

因為我的數據集實際有526列，所以這就是為什么我把它們分成兩個列表，一個包含95列名稱，這將是i ，其余431是我需要在行中顯示的列，如圖所示樣本數據集。

Answer 1

這將幫助您入門。 本質是使用set_index ，列轉換為MultiIndex，然后stack 。 可能存在更好的解決方案，但我會這樣做，因為它是輸出的簡單步驟。

# Set the index with columns that we don't want to "transpose"
df2 = df.set_index([
   'ID', 'Country', 'Type', 'Region', 'Gender', 'QA_Include', 'QA_Comments'])
# Convert headers to MultiIndex -- this is so we can melt IA values
df2.columns = pd.MultiIndex.from_tuples(map(tuple, df2.columns.str.split('_')))
# Call stack to replicate data, then reset the index
out =  df2.stack(level=0).reset_index().rename({'level_7': 'IA'}, axis=1)

out

     ID Country Type  Region  Gender QA_Include  QA_Comments    IA  Class1  Class2 Raw
0   SC1  France    A  Europe    Male        yes          NaN  IA01       8       1   4
1   SC1  France    A  Europe    Male        yes          NaN  IA02       4       1   J
2   SC2  France    A  Europe  Female        yes          NaN  IA01       7       2   2
3   SC2  France    A  Europe  Female        yes          NaN  IA02       6       4   Q
4   SC3  France    B  Europe    Male        yes          NaN  IA01       7       2   3
5   SC3  France    B  Europe    Male        yes          NaN  IA02       8       2   K
6   SC4  France    A  Europe    Male        yes          NaN  IA01       8       2   4
7   SC4  France    A  Europe    Male        yes          NaN  IA02       2       1   A
8   SC5  France    B  Europe    Male        yes          NaN  IA01       7       1   1
9   SC5  France    B  Europe    Male        yes          NaN  IA02       1       3   F
10  ID6  France    A  Europe    Male        yes          NaN  IA01       8       1   2
11  ID6  France    A  Europe    Male        yes          NaN  IA02       3       7   R
12  ID7  France    B  Europe    Male        yes          NaN  IA01       8       1   2
13  ID7  France    B  Europe    Male        yes          NaN  IA02       4       6   Q
14  UC8  France    B  Europe    Male        yes          NaN  IA01       8       2   4
15  UC8  France    B  Europe    Male        yes          NaN  IA02       4       2   P

Answer 2

你可以使用pd.lreshape

pd.lreshape(df.assign(IA01=['01']*len(df), IA02=['02']*len(df),IA09=['09']*len(df)), 
            {'IA': ['IA01', 'IA02','IA09'],
             'Raw': ['IA01_Raw','IA02_Raw','IA09_Raw'], 
             'Class1': ['IA01_Class1','IA02_Class1','IA09_Class1'], 
             'Class2': ['IA01_Class2', 'IA02_Class2','IA09_Class2']
             })


edit : 

pd.lreshape(df.assign(IA01=['01']*len(df), IA02=['02']*len(df),IA09=['09']*len(df)), 
            {'IA': ['IA01', 'IA02','IA09'],
             'Raw': ['IA01_Raw_baseline','IA02_Raw_midline','IA09_Raw_whatever'], 
             'Class1': ['IA01_Class1_baseline','IA02_Class1_midline','IA09_Class1_whatever'], 
             'Class2': ['IA01_Class2_baseline', 'IA02_Class2_midline','IA09_Class2_whatever']
             })

編輯：只需將輸出中所有column names從輸出的Raw/Class1/Class2列中的輸入添加到字典內的列表中

此文檔不可用。 使用help(pd.lreshape)或在這里參考

輸出：

    Country Gender  ID  QA_Comments QA_Include  Region  Type    IA  Raw Class1  Class2
0   France  Male    SC1 NaN         yes         Europe  A       01  4   8       1
1   France  Female  SC2 NaN         yes         Europe  A       01  2   7       2
2   France  Male    SC3 NaN         yes         Europe  B       01  3   7       2
3   France  Male    SC4 NaN         yes         Europe  A       01  4   8       2
4   France  Male    SC5 NaN         yes         Europe  B       01  1   7       1
5   France  Male    ID6 NaN         yes         Europe  A       01  2   8       1
6   France  Male    ID7 NaN         yes         Europe  B       01  2   8       1
7   France  Male    UC8 NaN         yes         Europe  B       01  4   8       2
8   France  Male    SC1 NaN         yes         Europe  A       02  J   4       1
9   France  Female  SC2 NaN         yes         Europe  A       02  Q   6       4
10  France  Male    SC3 NaN         yes         Europe  B       02  K   8       2
11  France  Male    SC4 NaN         yes         Europe  A       02  A   2       1
12  France  Male    SC5 NaN         yes         Europe  B       02  F   1       3
13  France  Male    ID6 NaN         yes         Europe  A       02  R   3       7
14  France  Male    ID7 NaN         yes         Europe  B       02  Q   4       6
15  France  Male    UC8 NaN         yes         Europe  B       02  P   4       2
16  France  Male    SC1 NaN         yes         Europe  A       09  W   6       3
17  France  Female  SC2 NaN         yes         Europe  A       09  X   5       2
18  France  Male    SC3 NaN         yes         Europe  B       09  Y   5       5
19  France  Male    SC4 NaN         yes         Europe  A       09  P   5       2
20  France  Male    SC5 NaN         yes         Europe  B       09  T   5       2
21  France  Male    ID6 NaN         yes         Europe  A       09  I   5       2
22  France  Male    ID7 NaN         yes         Europe  B       09  A   8       2
23  France  Male    UC8 NaN         yes         Europe  B       09  K   7       5