加快 python 性能

Question

adding 100k raw data and plot drawn by suggested code I am having below code, analyzing a 100K rows data, it takes 3 minutes for the output to be shown. the problem is with the for loops, where program needs to check two indicators and later act based upon that. data is a bourse buy/sell/na records and i want to draw buy vs sell and so on.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt


data = pd.read_csv('gco.csv', encoding ='utf-16 LE')
x = data.index.size
data['Money'] = data['Last']*data['Volume']
data['Date'] = data['Time']

# Creating date column
data['Date'] = data['Date'].map(lambda x: x[0:10])


# Creating a dedicated database
my_df = pd.DataFrame(columns =['Buy','Sell','NA'])

#calculate the Buy column
avai_dates = pd.unique(data.Date)
y = len(avai_dates)
my_df = pd.DataFrame(index=np.arange(0, y), columns =['Buy','Sell','NA'])
my_df[:]=0

for j in range(y):
    for i in range(x):
        if data.Date[i] == avai_dates[j] and data.Type[i] == 'Buy':
            my_df.Buy[j] += data.Money[i]
    for i in range(x):
        if data.Date[i] == avai_dates[j] and data.Type[i] == 'Sell':
            my_df.Sell[j] += data.Money[i]
    for i in range(x):
        if data.Date[i] == avai_dates[j] and data.Type[i] == 'Buy/Sell':
            my_df.NA[j] += data.Money[i]

new_df = my_df[(my_df.T != 0).any()]
z = len(new_df)
xm = np.arange(0, z)

plt.plot(xm, new_df.Buy, 'green')
plt.plot(xm, new_df.Sell, 'red')
plt.plot(xm, new_df.NA, 'yellow')
plt.xlabel('Dates', fontsize = 15)
plt.ylabel('Money Volumes', fontsize = 15)
plt.title('Buy vs. Sell Vs. NA')
plt.grid()
plt.show()

ax = plt.subplot(111)
ax.bar(xm-0.2, new_df.Buy, width = 0.2, color = 'g')
ax.bar(xm,new_df.Sell, width = 0.2 , color = 'r')
ax.bar(xm+0.2,new_df.NA, width = 0.2, color = 'y')

Answer 1

What you are doing is grouping the data points by Time and the Type and aggregating them. Pandas has build in functions for doing this.

You can replace all this code:

# Creating a dedicated database
my_df = pd.DataFrame(columns =['Buy','Sell','NA'])

#calculate the Buy column
avai_dates = pd.unique(data.Date)
y = len(avai_dates)
my_df = pd.DataFrame(index=np.arange(0, y), columns =['Buy','Sell','NA'])
my_df[:]=0

for j in range(y):
    for i in range(x):
        if data.Date[i] == avai_dates[j] and data.Type[i] == 'Buy':
            my_df.Buy[j] += data.Money[i]
    for i in range(x):
        if data.Date[i] == avai_dates[j] and data.Type[i] == 'Sell':
            my_df.Sell[j] += data.Money[i]
    for i in range(x):
        if data.Date[i] == avai_dates[j] and data.Type[i] == 'Buy/Sell':
            my_df.NA[j] += data.Money[i]

new_df = my_df[(my_df.T != 0).any()]

With this statement:

new_df = data.groupby(["Time", "Type"]).agg({'Money':['sum']})["Money","sum"].unstack(fill_value=0)

Breaking it down

data.groupby(["Time", "Type"])

This hierarchically groups the data by the Time and then by Type . For more information on this, check out the DataFrame.groupby() documentation.

.agg({'Money':['sum']})

This aggregates the Money values in each group by summing it up. You could just use .agg('sum') but this would also aggregate the values of 'Last' and 'Volume'

["Money","sum"]

Then we just unpack the columns to get to the raw sum. This gives you almost the result, however it has the Type group stacked:

Time        Type    
2020:12:12  Buy         1000
            Sell        1000
2020:12:13  Buy         4400
            Sell        2200
2020:12:14  Sell        4680
2020:12:15  Buy         2860
            Sell        1430
2020:12:16  Buy/Sell    6400
2020:12:17  Buy         7140
2020:12:18  Buy/Sell     770
2020:12:19  Buy          810
            Sell        1620
2020:12:20  Buy         2400
            Sell        1200
2020:12:21  Buy         1210
2020:12:22  Buy         1200
            Sell        1200
Name: (Money, sum), dtype: int64

You can now use the final unstacking function for getting the final dataframe. By setting fill_value=0 you ensure that the undefined values are set to 0 instead of nan

.unstack(fill_value=0)

I created some toy data based on the little info you have provided and running it through the one-liner, this is what you get.

Type         Buy  Buy/Sell  Sell
Time                            
2020:12:12  1000         0  1000
2020:12:13  4400         0  2200
2020:12:14     0         0  4680
2020:12:15  2860         0  1430
2020:12:16     0      6400     0
2020:12:17  7140         0     0
2020:12:18     0       770     0
2020:12:19   810         0  1620
2020:12:20  2400         0  1200
2020:12:21  1210         0     0
2020:12:22  1200         0  1200

It is basically almost identical to the original new_df you have computed, except that it keeps the Time values as index and Buy/Sell Type is labeled Buy/Sell instead of NA . Of course you can drop the Time column and rename the Buy/Sell if you so wish by appending this to the one-liner:

.reset_index().drop("Time",axis=1).rename(columns={"Buy/Sell":"NA"})

Let me know in the comments if this provides any speedup. If your data frame is really large, you might have to resort to other techniques such as analyzing the data in batches, parallel processing or custom numba processing.