Python numpy multidimensional array filtering by another array values

Question

I have a list consisting of 7 columns and over 43 thousand rows, representing x,y,z,R,G,B, Intensity of a point cloud from a LIDAR survey. I would like to filter the cloud by color, having manually isolated a portion. I would like to eliminate all the rows that do not have the RGB combination equal to one of those present in the filter. For example:

Data=[[4,8,12,179,118,74,I1], [5,10,45,142,186,98,I2], [7,14,21,185,193,112,I3], [8,16,24,115,140,43,I3]]

And I have a filter on columns 4, 5, and 6 (RGB), as follows:

Filter=[[179,118,74],[185,193,112]]

As output, I would like to get a new list containing only the items that meet the prerequisites (in this case they would be the first and third row). I would like a new list of this kind:

newlist=[[4,8,12,179,118,74,I1], [7,14,21,185,193,112,I3]]

I have no experience in Python, so I wouldn't know where to start.

Answer 1

There is one solution in comments:

Data[np.isin(Data[:, 3:6], Filter).all(axis=1)]

Indeed, np.isin is the most advisable option for you in such kind of problems.

I'm pretty sure it works faster if you reduce dimension of your Data and Filter before using np.isin in one of these ways:

def numpy_dimreduce_ravel(arr, shape):
    return np.ravel_multi_index(arr.T, shape, order='F')

def numpy_dimreduce_dot(arr, shape):
    dim_shape = np.cumprod(np.insert(shape[:-1], 0, 1))
    return np.dot(arr, dim_shape)
Data = np.array([[4,8,12,179,118,74], [5,10,45,142,186,98], [7,14,21,185,193,112], [8,16,24,115,140,43]])
shape = (256, 256, 256)

new_Data = numpy_dimreduce_ravel(data[:,3:6], shape) #[4880051, 6470286, 7389625, 2854003]
# or:
new_Data = numpy_dimreduce_dot(data[:,3:6], shape) #[4880051, 6470286, 7389625, 2854003]
# and:
new_Filter = numpy_dimreduce_ravel(Filter, shape) #[4880051, 7389625]

So you're able to do it in a more flexible way now:

>>> Data[np.isin(new_data, new_Filter)]
array([[  4,   8,  12, 179, 118,  74],
       [  7,  14,  21, 185, 193, 112]])