How to move data in numpy array from column/row to another based on value in third column

Question

Im trying to sort this data to go from this:

to this:

Basically I'm trying to compress 5 rows of data, each with 1 ID and 2 values into 1 row of data with 1 ID and 10 values. My data is approx. 6 million rows long. One thing to note: not every group has 5 (X,Y) coordinate values. Some only have 4.

I could not figure out how to do this by indexing alone. So i wrote a for loop, which doesnt work very well. It will sort the first 10,000 ok (but end with an error), but it takes forever.

coords = pd.read_csv('IDQQCoords.csv') 

coords = coords.as_matrix(columns=None) 

mpty = np.zeros((len(coords),8),dtype=float) 
#creates an empty array the same length as coords

coords = np.append(coords,mpty,axis=1) 
# adds the 8 empty columns from the previous command
#This is to make space to add the values from subsequent rows 



cnt = 0
lth = coords.shape[0]
for counter in range(1,lth):

    if coords[cnt+1,0] == coords[cnt,0]:
        coords[cnt,3:5] = coords[cnt+1,1:3]        
        coords = np.delete(coords,cnt+1,axis=0)

    if coords[cnt+1,0] == coords[cnt,0]:
        coords[cnt,5:7] = coords[cnt+1,1:3]       
        coords = np.delete(coords,cnt+1,axis=0)

    if coords[cnt+1,0] == coords[cnt,0]:
        coords[cnt,7:9] = coords[cnt+1,1:3]
        coords = np.delete(coords,cnt+1,axis=0)

    if coords[cnt+1,0] == coords[cnt,0]:
        coords[cnt,9:11] = coords[cnt+1,1:3]        
        coords = np.delete(coords,cnt+1,axis=0)

    cnt = cnt+1

Can someone help me, either with an index or a better loop?

Thanks a ton

Answer 1

Assuming that

coords = pd.read_csv('IDQQCoords.csv') 

implies that you are using Pandas, then the easiest way to produce the desired result is to use DataFrame.pivot :

import pandas as pd
import numpy as np
np.random.seed(2016)

df = pd.DataFrame({'shapeid': [0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 2],
               'x': np.random.random(14),
               'y': np.random.random(14)}) 
#     shapeid         x         y
# 0         0  0.896705  0.603638
# 1         0  0.730239  0.588791
# 2         0  0.783276  0.069347
# 3         0  0.741652  0.942829
# 4         0  0.462090  0.372599
# 5         1  0.642565  0.451989
# 6         1  0.224864  0.450841
# 7         1  0.708547  0.033112
# 8         1  0.747126  0.169423
# 9         2  0.625107  0.180155
# 10        2  0.579956  0.352746
# 11        2  0.242640  0.342806
# 12        2  0.131956  0.277638
# 13        2  0.143948  0.375779

df['col'] = df.groupby('shapeid').cumcount()
df = df.pivot(index='shapeid', columns='col')
df = df.sort_index(axis=1, level=1)
df.columns = ['{}{}'.format(col, num) for col,num in df.columns]
print(df)

yields

               x0        y0        x1        y1        x2        y2        x3  \
shapeid                                                                         
0        0.896705  0.603638  0.730239  0.588791  0.783276  0.069347  0.741652   
1        0.642565  0.451989  0.224864  0.450841  0.708547  0.033112  0.747126   
2        0.625107  0.180155  0.579956  0.352746  0.242640  0.342806  0.131956   

               y3        x4        y4  
shapeid                                
0        0.942829  0.462090  0.372599  
1        0.169423       NaN       NaN  
2        0.277638  0.143948  0.375779