Pandas DataFrame 耗时太长

Question

I am running the below code on a file with close to 300k lines. I know my code is not very efficient as it takes forever to finish, can anyone advise me on how I can speed it up?

import sys
import numpy as np
import pandas as pd


file = sys.argv[1]

df = pd.read_csv(file, delimiter=' ',header=None)
df.columns = ["ts", "proto", "orig_bytes", "orig_pkts", "resp_bytes", "resp_pkts", "duration", "conn_state"]

orig_bytes = np.array(df['orig_bytes'])
resp_bytes = np.array(df['resp_bytes'])


size = np.array([])
ts = np.array([])
for i in range(len(df)):
    if orig_bytes[i] > resp_bytes[i]:
        size = np.append(size, orig_bytes[i])
        ts = np.append(ts, df['ts'][i])
    else:
        size = np.append(size, resp_bytes[i])
        ts = np.append(ts, df['ts'][i])

The aim is to only record instances where one of the two (orig_bytes or resp_bytes) is the larger one.

Thanking you all for your help

Answer 1

I can't guarantee that this will run faster than what you have, but it is a more direct route to where you want to go. Also, I'm assuming based on your example that you don't want to keep instances where the two byte values are equal and that you want a separate DataFrame in the end, not a new column in the existing df:

After you've created your DataFrame and renamed the columns, you can use query to drop all the instances where orig_bytes and resp_bytes are the same, create a new column with the max value of the two, and then narrow the DataFrame down to just the two columns you want.

df = pd.read_csv(file, delimiter=' ',header=None)
df.columns = ["ts", "proto", "orig_bytes", "orig_pkts", "resp_bytes", "resp_pkts", "duration", "conn_state"]

df_new = df.query("orig_bytes != resp_bytes")
df_new['biggest_bytes'] = df_new[['orig_bytes', 'resp_bytes']].max(axis=1)
df_new = df_new[['ts', 'biggest_bytes']]

If you do want to include the entries where they are equal to each other, then just skip the query step.