堆棧內存溢出
  簡體   English   中英

如何對復雜的 dataframe 結構進行非規范化？

[英]How to denormalize a complex dataframe structure?

 原文  2021-03-13 09:11:18  5   2    python/ pandas/ pivot

問題描述

我有一個看起來像這樣的數據框：

         USUBJID   IDVAR IDVARVAL      QNAM                        QVAL
0  Dummy-01-0001   AESEQ      1.0  AEdummy1                  2012-02-15
1  Dummy-01-0002   AESEQ      1.0  AEdummy1                  2012-02-23
2  Dummy-01-0004   AESEQ      1.0  AEdummy1                  2012-02-06
3  Dummy-01-0004   AESEQ      2.0  AEdummy1                  2012-03-10
4  Dummy-01-0005   AESEQ      1.0  AEdummy1                  2012-03-10
5  Dummy-01-0001  AESPID        1  dummy2AE  Gastrointestinal disorders
6  Dummy-01-0002  AESPID        1  dummy2AE     Nervous system disorder
7  Dummy-01-0004  AESPID        2  dummy2AE  Gastrointestinal disorders
8  Dummy-01-0004  AESPID        1  dummy2AE     Nervous system disorder
9  Dummy-01-0005  AESPID        1  dummy2AE  Gastrointestinal disorders

以上 dataframe 只是一個示例，請使用以下代碼獲取更多數據到 dataframe。

df = pd.DataFrame({'USUBJID': {0: 'Dummy-01-0001', 1: 'Dummy-01-0002', 2: 'Dummy-01-0004', 3: 'Dummy-01-0004', 4: 'Dummy-01-0005', 5: 'Dummy-01-0007', 6: 'Dummy-01-0008', 7: 'Dummy-01-0008', 8: 'Dummy-01-0010', 9: 'Dummy-01-0010', 10: 'Dummy-01-0011', 11: 'Dummy-01-0011', 12: 'Dummy-01-0013', 13: 'Dummy-01-0013', 14: 'Dummy-01-0014', 15: 'Dummy-01-0016', 16: 'Dummy-01-0016', 17: 'Dummy-01-0019', 18: 'Dummy-01-0001', 19: 'Dummy-01-0002', 20: 'Dummy-01-0004', 21: 'Dummy-01-0004', 22: 'Dummy-01-0005', 23: 'Dummy-01-0007', 24: 'Dummy-01-0008', 25: 'Dummy-01-0008', 26: 'Dummy-01-0010', 27: 'Dummy-01-0010', 28: 'Dummy-01-0011', 29: 'Dummy-01-0011', 30: 'Dummy-01-0013', 31: 'Dummy-01-0013', 32: 'Dummy-01-0014', 33: 'Dummy-01-0016', 34: 'Dummy-01-0016', 35: 'Dummy-01-0017', 36: 'Dummy-01-0017', 37: 'Dummy-01-0019'}, 'IDVAR': {0: 'AESEQ', 1: 'AESEQ', 2: 'AESEQ', 3: 'AESEQ', 4: 'AESEQ', 5: 'AESEQ', 6: 'AESEQ', 7: 'AESEQ', 8: 'AESEQ', 9: 'AESEQ', 10: 'AESEQ', 11: 'AESEQ', 12: 'AESEQ', 13: 'AESEQ', 14: 'AESEQ', 15: 'AESEQ', 16: 'AESEQ', 17: 'AESEQ', 18: 'AESPID', 19: 'AESPID', 20: 'AESPID', 21: 'AESPID', 22: 'AESPID', 23: 'AESPID', 24: 'AESPID', 25: 'AESPID', 26: 'AESPID', 27: 'AESPID', 28: 'AESPID', 29: 'AESPID', 30: 'AESPID', 31: 'AESPID', 32: 'AESPID', 33: 'AESPID', 34: 'AESPID', 35: 'AESPID', 36: 'AESPID', 37: 'AESPID'}, 'IDVARVAL': {0: '1.0', 1: '1.0', 2: '1.0', 3: '2.0', 4: '1.0', 5: '1.0', 6: '1.0', 7: '2.0', 8: '1.0', 9: '2.0', 10: '1.0', 11: '2.0', 12: '1.0', 13: '2.0', 14: '1.0', 15: '1.0', 16: '2.0', 17: '1.0', 18: '1', 19: '1', 20: '2', 21: '1', 22: '1', 23: '1', 24: '1', 25: '2', 26: '2', 27: '1', 28: '1', 29: '2', 30: '1', 31: '2', 32: '1', 33: '1', 34: '2', 35: '1', 36: '2', 37: '1'}, 'QNAM': {0: 'AEdummy1', 1: 'AEdummy1', 2: 'AEdummy1', 3: 'AEdummy1', 4: 'AEdummy1', 5: 'AEdummy1', 6: 'AEdummy1', 7: 'AEdummy1', 8: 'AEdummy1', 9: 'AEdummy1', 10: 'AEdummy1', 11: 'AEdummy1', 12: 'AEdummy1', 13: 'AEdummy1', 14: 'AEdummy1', 15: 'AEdummy1', 16: 'AEdummy1', 17: 'AEdummy1', 18: 'dummy2AE', 19: 'dummy2AE', 20: 'dummy2AE', 21: 'dummy2AE', 22: 'dummy2AE', 23: 'dummy2AE', 24: 'dummy2AE', 25: 'dummy2AE', 26: 'dummy2AE', 27: 'dummy2AE', 28: 'dummy2AE', 29: 'dummy2AE', 30: 'dummy2AE', 31: 'dummy2AE', 32: 'dummy2AE', 33: 'dummy2AE', 34: 'dummy2AE', 35: 'dummy2AE', 36: 'dummy2AE', 37: 'dummy2AE'}, 'QVAL': {0: '2012-02-15', 1: '2012-02-23', 2: '2012-02-06', 3: '2012-03-10', 4: '2012-03-10', 5: '2012-02-08', 6: '2012-03-18', 7: '2012-03-07', 8: '2012-02-01', 9: '2012-01-10', 10: '2012-01-19', 11: '2012-03-28', 12: '2012-02-19', 13: '2012-02-14', 14: '2012-03-13', 15: '2012-03-08', 16: '2012-02-05', 17: '2012-03-18', 18: 'Gastrointestinal disorders', 19: 'Nervous system disorder', 20: 'Gastrointestinal disorders', 21: 'Nervous system disorder', 22: 'Gastrointestinal disorders', 23: 'Vascular disorders', 24: 'Gastrointestinal disorders', 25: 'Vascular disorders', 26: 'Nervous system disorder', 27: 'Gastrointestinal disorders', 28: 'Nervous system disorder', 29: 'Nervous system disorder', 30: 'Nervous system disorder', 31: 'Gastrointestinal disorders', 32: 'Gastrointestinal disorders', 33: 'Vascular disorders', 34: 'Gastrointestinal disorders', 35: 'Nervous system disorder', 36: 'Nervous system disorder', 37: 'Vascular disorders'}})

讓我們稍微了解一下結構。 IDVAR保存變量名， IDVARVAL保存其值， QNAM保存另一個變量名， QVAL保存相應的值。 在我正在工作的領域中，這種結構稱為規范化結構，但我知道它很奇怪。

我想以下列形式獲取這些數據：

         USUBJID   AESEQ AESPID    AEdummy1      dummy2AE
0  Dummy-01-0001   1.0      1      2012-02-15    Gastrointestinal disorders
1  Dummy-01-0002   1.0      1      2012-02-23    Nervous system disorder
2  Dummy-01-0004   1.0      2      2012-02-06    Gastrointestinal disorders
3  Dummy-01-0004   2.0      1      2012-03-10    Nervous system disorder
4  Dummy-01-0005   1.0      1      2012-03-10    Gastrointestinal disorders

如果我只需要為QNAM和QVAL做這件事，沒有問題，我可以使用 pandas 的 pivot 輕松完成，如下所示：

df.pivot(
        index=['USUBJID', 'IDVARVAL', 'IDVAR'], 
         columns='QNAM', 
         values='QVAL'
).reset_index()

即使對於這個問題，我也可以使用某種屏蔽，但我知道這不會是最有效的方法，因為這些數據將有數千甚至數百萬條記錄。

請注意： IDVAR 的USUBJID和IDVARVAL的IDVAR應該 map 到QVAL的QNAM的值。 話雖如此，對於AESEQ DataFrame， AESPID 、 AEdummy1 、AEdummy1 和dummy2AE的一些記錄可能是 NaN 或空。 換言之，對於上述樣本， Dummy-01-0001和AESEQ=1.0唯一標識AEdummy1 = 2012-02-15 。

2 個解決方案

解決方案1
 6 已采納  2021-03-13 09:30:07

讓我們嘗試unstacking后跟concat ：

s  = df.set_index([df.groupby('IDVAR').cumcount(), 'USUBJID'])
s1 = s.set_index('IDVAR', append=True)['IDVARVAL'].unstack()
s2 = s.set_index('QNAM',  append=True)['QVAL'].unstack()

out = pd.concat([s1, s2], axis=1).reset_index(level=1)

解釋

在 IDVAR 上對IDVAR進行分組，並使用cumcount創建一個順序計數器來唯一標識每個IDVAR組的行，然后將此計數器與列USUBJID一起設置為 dataframe 的索引：

>>> s
                  IDVAR  IDVARVAL      QNAM                        QVAL
  USUBJID                                                              
0 Dummy-01-0001   AESEQ       1.0  AEdummy1                  2012-02-15
1 Dummy-01-0002   AESEQ       1.0  AEdummy1                  2012-02-23
2 Dummy-01-0004   AESEQ       1.0  AEdummy1                  2012-02-06
3 Dummy-01-0004   AESEQ       2.0  AEdummy1                  2012-03-10
4 Dummy-01-0005   AESEQ       1.0  AEdummy1                  2012-03-10
0 Dummy-01-0001  AESPID       1.0  dummy2AE  Gastrointestinal disorders
1 Dummy-01-0002  AESPID       1.0  dummy2AE     Nervous system disorder
2 Dummy-01-0004  AESPID       2.0  dummy2AE  Gastrointestinal disorders
3 Dummy-01-0004  AESPID       1.0  dummy2AE     Nervous system disorder
4 Dummy-01-0005  AESPID       1.0  dummy2AE  Gastrointestinal disorders

現在，對於每列IDVARVAL和QVAL ，通過附加相應的列IDVAR和QNAM來更新索引，然后取消unstack以重塑：

>>> s1

  IDVAR          AESEQ  AESPID
  USUBJID                     
0 Dummy-01-0001    1.0     1.0
1 Dummy-01-0002    1.0     1.0
2 Dummy-01-0004    1.0     2.0
3 Dummy-01-0004    2.0     1.0
4 Dummy-01-0005    1.0     1.0

>>> s2

  QNAM             AEdummy1                    dummy2AE
  USUBJID                                              
0 Dummy-01-0001  2012-02-15  Gastrointestinal disorders
1 Dummy-01-0002  2012-02-23     Nervous system disorder
2 Dummy-01-0004  2012-02-06  Gastrointestinal disorders
3 Dummy-01-0004  2012-03-10     Nervous system disorder
4 Dummy-01-0005  2012-03-10  Gastrointestinal disorders

最后，將上述未堆疊的幀concat和s2沿axis=1 s1以獲得所需的結果：

>>> out

         USUBJID  AESEQ  AESPID    AEdummy1                    dummy2AE
0  Dummy-01-0001    1.0     1.0  2012-02-15  Gastrointestinal disorders
1  Dummy-01-0002    1.0     1.0  2012-02-23     Nervous system disorder
2  Dummy-01-0004    1.0     2.0  2012-02-06  Gastrointestinal disorders
3  Dummy-01-0004    2.0     1.0  2012-03-10     Nervous system disorder
4  Dummy-01-0005    1.0     1.0  2012-03-10  Gastrointestinal disorders

解決方案2
 5  2021-03-13 10:24:58

您可以使用pyjanitor 中的 pivot_longer 重塑數據，使用 cumcount 創建一個計數器以創建唯一索引，pivot數據， 並進行一些最終清理以獲得類似於您預期的 output 的內容：

(
    df.pivot_longer(["USUBJID"],
                    names_pattern=["IDVAR$|QNAM$", "IDVARVAL$|QVAL$"],
                    names_to=["variables", "values"])
    .assign(counter=lambda df: df.groupby(["USUBJID", "variables"]).cumcount())
    .pivot(["USUBJID", "counter"], "variables", "values")
    .droplevel("counter")
    .rename_axis(columns=None)
    .reset_index()
)

         USUBJID AESEQ AESPID    AEdummy1                    dummy2AE
0  Dummy-01-0001   1.0    1.0  2012-02-15  Gastrointestinal disorders
1  Dummy-01-0002   1.0    1.0  2012-02-23     Nervous system disorder
2  Dummy-01-0004   1.0    2.0  2012-02-06  Gastrointestinal disorders
3  Dummy-01-0004   2.0    1.0  2012-03-10     Nervous system disorder
4  Dummy-01-0005   1.0    1.0  2012-03-10  Gastrointestinal disorders

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 如何對熊貓數據框進行非規范化 如何為 Pandas Dataframe 非規范化 YAML？ 如何將多索引數據幀轉換為復雜結構？ 如何從具有復雜結構的列表中創建 DataFrame？ 來自復雜數據結構的數據框 如何構造復雜的查詢 Pandas加載jsonl到dataframe內部結構復雜 如何使火炬張量“非正規化”？ 如何調整復雜的數據框架 如何構建復雜的Django項目？
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM