python-为800万行数据帧逐行填充唯一值的最快时间

Question

I have a dataframe named report1 ; with the size of 14 colums X 8 million rows. What I would like to do is to get unique values of each row from column 3 through to column 8 and populate the result of each row onto a new dataframe named df .

report1 (source data) looks like below:

Ticket Number Col0 Col1 Col2 Col3 Col4 Col5 Col6 Col7 Col8

100 21 30 32 3 4 6 1 5 0

101 4 9 25 3 4 6 1 5 4

102 45 33 11 3 4 6 1 5 3

… … … … … … … … … …

8000000 12 5 28 3 4 6 1 5 11

df (new dataframe)should be like this:

Ticket Number Col0 Col1 Col2 Col3 Col4 Col5 Col6 Col7 Col8

100 21 30 32 3 4 6 1 5 0

101 4 9 25 3 4 6 1 5 nan

102 45 33 11 3 4 6 1 5 nan

… … … … … … … … … …

8000000 12 5 28 3 4 6 1 5 11

So far I have been able to get what I wanted from the simple script below but it just takes too long to run it through even if I have tried get it run under pythonanywhere platform.

Anyone knows how to get this done in the shortest possible time?

The script is as follows:

result = []
for i in range(0,7999999):
    g = pd.unique(report1.iloc[i,7:13].values.ravel())

    arr_list = g.tolist()
    result.append(arr_list)

df = pd.DataFrame(result)
df

Answer 1

You need numpy :

data = report1.iloc[:,4:10].values
sort_idx = np.argsort(data,axis=1)
row_offset = data.shape[1]*np.arange(data.shape[0])[:,None]
sort_lin_idx = sort_idx[:,1::] + row_offset
dup_lin_idx = sort_lin_idx[np.diff(np.sort(data,axis=1),axis=1)==0]
a = data.ravel().astype(float)
a[dup_lin_idx] = np.nan
data = a.reshape(len(data), -1)
print (pd.DataFrame(data))

     0    1    2    3    4     5
0  3.0  4.0  6.0  1.0  5.0   0.0
1  3.0  4.0  6.0  1.0  5.0   NaN
2  3.0  4.0  6.0  1.0  5.0   NaN
3  3.0  4.0  6.0  1.0  5.0  11.0

Timings :

In [117]: %timeit (orig(report1))
1 loop, best of 3: 7.48 s per loop

In [118]: %timeit (jez1(report1))
1 loop, best of 3: 4.82 s per loop

In [119]: %timeit (jez2(report1))
100 loops, best of 3: 9.57 ms per loop

Code for timings :

#[40000 rows x 6 columns]
report1 = pd.concat([report1]*10000).reset_index(drop=True)

def orig(df):
    result = []
    for i in range(len(df.index)):
        g = pd.unique(report1.iloc[i,4:10].values.ravel())
        arr_list = g.tolist()
        result.append(arr_list)
    df = pd.DataFrame(result)
    return (df)

def jez1(df):
    df = report1.iloc[:,4:10]
    return (df.where(~df.apply(pd.Series.duplicated, axis=1), np.nan))

def jez2(report1):
    data = report1.iloc[:,4:10].values
    sort_idx = np.argsort(data,axis=1)
    row_offset = data.shape[1]*np.arange(data.shape[0])[:,None]
    sort_lin_idx = sort_idx[:,1::] + row_offset
    dup_lin_idx = sort_lin_idx[np.diff(np.sort(data,axis=1),axis=1)==0]
    a = data.ravel().astype(float)
    a[dup_lin_idx] = np.nan
    data = a.reshape(len(data), -1)
    return (pd.DataFrame(data))

print (orig(report1))    
print (jez1(report1))
print (jez2(report1))