Most of the entries in encoder output of an AutoEncoder is zero while used for feature extraction from high-resolution images

Question

I am using a simple Autoencoder to extract meaningful features from high-resolution images (2K).

My simple AutoEncoder looks like:

encoding_dim = 20
encoded = tf.keras.layers.Dense(encode_dim, activation='relu')(input)
decoded = tf.keras.layers.Dense(shape_img_flattened[0], activation='sigmoid')(encoded)

Models:

autoencoder = tf.keras.Model(input, decoded)
encoder = tf.keras.Model(input, encoded)

fitting the end-2-end autoencoder model:

autoencoder.fit(X_train, X_train,epochs = n_epochs,batch_size = batch_size,shuffle = True)

With this encoding_dimension or intermediate hidden size, most of the entries in intermediate representation or encoder output are zero.

(encoder_output = encoder.predict(X_test))

Can anyone please tell me why most of the entries in encoder output or prediction are zero (ie doesn't provide any meaningful features as such)?

Answer 1

Yes This is because you are using Relu as an activation function. Change that to LeakyRelu or sigmoid then it will work fine.