How to build this list comprehension correctly:

Question

How should I write this list comprehension properly: I am trying to keep from the train set X_tr only images for which numpy arrays have a.std() above 10 but I need to keep track of the index to return the corresponding labels in y_tr:

(X_tr,y_tr) = (np.array([i[i.std() >= 10] for i in X_tr.numpy()])[0],y_tr[i.iloc()].numpy())

Why is the error message telling me i is an int, i is supposed to be a image from the train set X_tr as a numpy object?

AttributeError                            Traceback (most recent call last)
<ipython-input-38-b9a613449db3> in <module>
----> 1 (X_tr,y_tr) = (np.array([i[i.std() >= 10] for i in X_tr.numpy()])[0],y_tr[i.iloc()].numpy())

AttributeError: 'int' object has no attribute 'iloc'

I also tried:

indexlist = []
index = 0

X_tr = X_tr.numpy()
X_val = X_val.numpy()
y_tr = y_tr.numpy()
y_val = y_val.numpy()

for i in range(len(X_tr)):
    if X_tr[i].std()<10:
        print(X_tr[i])
        indexlist.append(index)
    index += 1
    
X_tr = np.delete(X_tr, indexlist).reshape(15000 - len(indexlist), 65,65,7)
y_tr = np.delete(y_tr, indexlist).reshape(15000 - len(indexlist), 1,4)
indexlist = []
index = 0
for i in range (len(X_val)):
    if X_val[i].std()<10:
        indexlist.append(index)
    index += 1
X_val = np.delete(X_val, indexlist).reshape(5000 - len(indexlist), 65,65,7)
y_val = np.delete(y_val, indexlist).reshape(5000 - len(indexlist), 1,4)


 [[0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  ...
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]]]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-47-02ccbdb56efd> in <module>
     13     index += 1
     14 
---> 15 X_tr = np.delete(X_tr, indexlist).reshape(15000 - len(indexlist), 65,65,7)
     16 y_tr = np.delete(y_tr, indexlist).reshape(15000 - len(indexlist), 1,4)
     17 indexlist = []

ValueError: cannot reshape array of size 443624848 into shape (14848,65,65,7)

and:

indices = np.where(np.std(X_tr, axis=1) > 10)
X_tr, y_tr = X_tr[indices], y_tr[indices]

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-45e827a14263> in <module>
      1 indices = np.where(np.std(X_tr, axis=1) > 10)
----> 2 X_tr, y_tr = X_tr[indices], y_tr[indices]

~/.pyenv/versions/lewagon/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
    199     """Call target, and fall back on dispatchers if there is a TypeError."""
    200     try:
--> 201       return target(*args, **kwargs)
    202     except (TypeError, ValueError):
    203       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~/.pyenv/versions/lewagon/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py in _slice_helper(tensor, slice_spec, var)
    984       new_axis_mask |= (1 << index)
    985     else:
--> 986       _check_index(s)
    987       begin.append(s)
    988       end.append(s + 1)

~/.pyenv/versions/lewagon/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py in _check_index(idx)
    863     # TODO(slebedev): IndexError seems more appropriate here, but it
    864     # will break `_slice_helper` contract.
--> 865     raise TypeError(_SLICE_TYPE_ERROR + ", got {!r}".format(idx))
    866 
    867 

TypeError: Only integers, slices (`:`), ellipsis (`...`), tf.newaxis (`None`) and scalar tf.int32/tf.int64 tensors are valid indices, got array([    0,     0,     0, ..., 14999, 14999, 14999])

and:

indices = np.where(np.std(X_tr.numpy(), axis=1) > 10)
X_tr, y_tr = X_tr.numpy(), y_tr.numpy()
X_tr, y_tr = X_tr[indices], y_tr[indices]

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-51-6a9c5ffab9a9> in <module>
      1 indices = np.where(np.std(X_tr.numpy(), axis=1) > 10)
      2 X_tr, y_tr = X_tr.numpy(), y_tr.numpy()
----> 3 X_tr, y_tr = X_tr[indices], y_tr[indices]

IndexError: too many indices for array

My X_tr looks like this, for the moment np.where doesn't work:

<tf.Tensor: shape=(15000, 65, 65, 7), dtype=uint8, numpy=
array([[[[ 25,  25,  28, ...,  65,  47,  26],
         [ 25,  25,  27, ...,  61,  43,  23],
         [ 25,  26,  30, ...,  67,  49,  23],
         ...,
         [ 28,  29,  31, ...,  77,  54,  28],
         [ 30,  32,  36, ...,  81,  58,  31],
         [ 30,  33,  38, ...,  84,  60,  34]],

        [[ 24,  23,  25, ...,  56,  37,  25],
         [ 24,  22,  24, ...,  59,  38,  23],
         [ 26,  26,  33, ...,  72,  51,  25],
         ...,
         [ 28,  30,  32, ...,  76,  52,  28],
         [ 29,  31,  34, ...,  79,  54,  29],
         [ 29,  32,  35, ...,  79,  57,  31]],

        [[ 27,  28,  34, ...,  70,  52,  31],
         [ 25,  25,  29, ...,  62,  45,  30],
         [ 26,  27,  34, ...,  73,  54,  26],
         ...,
         [ 26,  27,  28, ...,  71,  47,  25],
         [ 28,  30,  33, ...,  76,  52,  27],
         [ 29,  31,  36, ...,  79,  55,  32]],

        ...,

        [[ 29,  29,  35, ...,  68,  50,  32],
         [ 28,  28,  34, ...,  72,  53,  31],
         [ 28,  29,  33, ...,  64,  49,  31],
         ...,
         [ 33,  36,  44, ...,  83,  61,  40],
         [ 32,  35,  41, ...,  80,  58,  36],
         [ 32,  35,  40, ...,  79,  56,  37]],

Answer 1

The purpose of list comprehensions is to make code clearer and cleaner. Your code is nearly impossible to understand. I'd separate calculating the indices.

indices = np.where(np.std(x, axis=1) > 10)
x, y = x[indices], y[indices]

I think that's what you're trying to do.