How to select records randomly from the DataFrame?

Question

I have the following single column pandas DataFrame called y . The column is called 0 (zero).

y =

1
0
0
1
0
1
1
2
0
1
1
2
2
2
2
1
0
0

I want to select N row indices of records per y value. In the above example, there are 6 records of 0 , 7 records of 1 and 5 records of 2 . I need to select 4 records from each of these 3 groups.

Below I provide my code. However this code always selects the first N (eg 4) records per class. I need the selection to be done randomly on a whole dataset.

How can I do it?

idx0 = []
idx1 = []
idx2 = []

for i in range(0, len(y[0])):
    if y[0].iloc[i]==0 and len(idx0)<=4:
        idx0.append(i)
    if y[0].iloc[i]==1 and len(idx1)<=4:
        idx1.append(i)
    if y[0].iloc[i]==2 and len(idx2)<=4:
        idx2.append(i)

Update:

The expected outcome is a list of indices, not the filtered DataFrame y .

n = 4
a = y.groupby(0).apply(lambda x: x.sample(n)).reset_index(1).\
    rename(columns={'level_1':'indices'}).reset_index(drop=True).groupby(0)['indices'].\
                                                    apply(list).reset_index()

class = 0
idx = a.ix[2].tolist()[class]
y.values[idx]   # THIS RETURNS WRONG WRONG CLASSES IN SOME CASES

0
1. # <- WRONG
0
0

Answer 1

Use groupby() with df.sample() :

n=4
df.groupby('Y').apply(lambda x: x.sample(n)).reset_index(drop=True)

    Y
0   0
1   0
2   0
3   0
4   1
5   1
6   1
7   1
8   2
9   2
10  2
11  2

EDIT, for index:

df.groupby('Y').apply(lambda x: x.sample(n)).reset_index(1).\
    rename(columns={'level_1':'indices'}).reset_index(drop=True).groupby('Y')['indices'].\
                                                    apply(list).reset_index()

   Y          indices
0  0   [4, 1, 17, 16]
1  1    [0, 6, 10, 5]
2  2  [13, 14, 7, 11]

Answer 2

Using

idx0,idx1,idx2=[ np.random.choice(y.index.values,4,replace=False).tolist()for _, y in df.groupby('0')]
idx0
Out[48]: [1, 2, 16, 8]

---

To be more detail

s=pd.Series([1,0,1,0,2],index=[1,3,4,5,9])
idx=[1,4] # both anky and mine answer return the index
s.loc[idx] # using .loc with index is correct 
Out[59]: 
1    1
4    1
dtype: int64
s.values[idx]# using value with slice with index, is wrong
Out[60]: array([0, 2], dtype=int64)

Answer 3

Supposing column "y" belongs to a dataframe "df" and you want to select N=4 random rows:

for i in np.unique(df.y).astype(int):
    print(df.y[np.random.choice(np.where(df.y==np.unique(df.y)[i])[0],4)])

You will get:

10116    0
329      0
4709     0
5630     0
Name: y, dtype: int32
382     1
392     1
9124    1
383     1
Name: y, dtype: int32
221      2
443      2
4235     2
5322     2
Name: y, dtype: int32

Edited , to get index:

pd.concat([df.y[np.random.choice(np.where(df.y==np.unique(df.y)[i])[0],4)] for i in np.unique(df.y).astype(int)],axis=0)

You will get:

10116    0
329      0
4709     0
5630     0
382      1
392      1
9124     1
383      1
221      2
443      2
4235     2
5322     2
Name: y, dtype: int32

To get a nested list of indices:

[df.holiday[np.random.choice(np.where(df.holiday==np.unique(df.holiday)[i])[0],4)].index.tolist() for i in np.unique(df.holiday).astype(int)]

You will get:

[[10116,329,4709,5630],[382,392,9124,383],[221,443,4235,5322]]

Answer 4

N = 4
y.loc[y[0]==0].sample(N)
y.loc[y[0]==1].sample(N)
y.loc[y[0]==2].sample(N)