向現有熊貓數據框添加行的最快方法

Question

我目前正在嘗試根據現有的csv創建一個新的csv。

我找不到基於現有數據框值設置數據框值的更快方法。

import pandas
import sys
import numpy
import time

# path to file as argument
path = sys.argv[1]
df = pandas.read_csv(path, sep = "\t")

# only care about lines with response_time
df = df[pandas.notnull(df['response_time'])]

# new empty dataframe
new_df = pandas.DataFrame(index = df["datetime"])    

# new_df needs to have datetime as index 
# and columns based on a combination 
# of 2 columns name from previous dataframe 
# (there are only 10 differents combinations)
# and response_time as values, so there will be lots of 
# blank cells but I don't care
for i, row in df.iterrows():
    start = time.time()
    new_df.set_value(row["datetime"], row["name"] + "-" + row["type"], row["response_time"])
    print(i, time.time() - start)

原始數據框為：

                     datetime           name   type  response_time
0  2018-12-18T00:00:00.500829    HSS_ANDROID  audio        0.02430
1  2018-12-18T00:00:00.509108    HSS_ANDROID  video        0.02537
2  2018-12-18T00:00:01.816758       HSS_TEST  audio        0.03958
3  2018-12-18T00:00:01.819865       HSS_TEST  video        0.03596
4  2018-12-18T00:00:01.825054  HSS_ANDROID_2  audio        0.02590
5  2018-12-18T00:00:01.842974  HSS_ANDROID_2  video        0.03643
6  2018-12-18T00:00:02.492477    HSS_ANDROID  audio        0.01575
7  2018-12-18T00:00:02.509231    HSS_ANDROID  video        0.02870
8  2018-12-18T00:00:03.788196       HSS_TEST  audio        0.01666
9  2018-12-18T00:00:03.807682       HSS_TEST  video        0.02975

new_df將如下所示：

我每個循環需要7毫秒。

處理（僅？個）400 000行數據框需要很長時間。 我怎樣才能使其更快？

Answer 1

確實，使用pivot將滿足您的需求，例如：

import pandas as pd
new_df = pd.pivot(df.datetime, df.name + '-' + df.type, df.response_time)
print (new_df.head())
                           HSS_ANDROID-audio  HSS_ANDROID-video  \
datetime                                                           
2018-12-18T00:00:00.500829             0.0243                NaN   
2018-12-18T00:00:00.509108                NaN            0.02537   
2018-12-18T00:00:01.816758                NaN                NaN   
2018-12-18T00:00:01.819865                NaN                NaN   
2018-12-18T00:00:01.825054                NaN                NaN   

                            HSS_ANDROID_2-audio  HSS_ANDROID_2-video  \
datetime                                                               
2018-12-18T00:00:00.500829                  NaN                  NaN   
2018-12-18T00:00:00.509108                  NaN                  NaN   
2018-12-18T00:00:01.816758                  NaN                  NaN   
2018-12-18T00:00:01.819865                  NaN                  NaN   
2018-12-18T00:00:01.825054               0.0259                  NaN   

                            HSS_TEST-audio  HSS_TEST-video  
datetime                                                    
2018-12-18T00:00:00.500829             NaN             NaN  
2018-12-18T00:00:00.509108             NaN             NaN  
2018-12-18T00:00:01.816758         0.03958             NaN  
2018-12-18T00:00:01.819865             NaN         0.03596  
2018-12-18T00:00:01.825054             NaN             NaN  

並且沒有NaN ，您可以將fillna與所需的任何值一起使用，例如：

new_df = pd.pivot(df.datetime, df.name +'-'+df.type, df.response_time).fillna(0)

Answer 2

您也可以使用unstack作為另一種選擇

new = df.set_index(['type','name', 'datetime']).unstack([0,1])
new.columns = ['{}-{}'.format(z,y) for x,y,z, in new.columns]

使用f-strings會比format快一點：

new.columns = [f'{z}-{y}' for x,y,z, in new.columns]