Proper way to evaluate classification model ('UndefinedMetricWarning:')

Question

I am currently using a neural network that outputs a one hot encoded output.

Upon evaluating it with a classification report I am receiving this error:

UndefinedMetricWarning: Recall and F-score are ill-defined and being set 
to 0.0 in samples with no true labels.

When one-hot encoding my output during the train-test-split phase, I had to drop one of the columns in order to avoid the Dummy Variable Trap. As a result, some of the predictions of my neural network are [0, 0, 0, 0] , signaling that it belongs to the fifth category. I believe this to be the cause of the UndefinedMetricWarning: .

Is there a solution to this? Or should I avoid classification reports in the first place and is there a better way to evaluate these sorts of neural networks? I'm fairly new to machine learning and neural networks, please forgive my ignorance. Thank you for all the help!!

---

Edit #1:

Here is my network:

from keras.models import Sequential
from keras.layers import Dense

classifier = Sequential()
classifier.add(Dense(units = 10000,
                     input_shape = (30183,),
                     kernel_initializer = 'glorot_uniform',
                     activation = 'relu'
                    )
              )
classifier.add(Dense(units = 4583,
                     kernel_initializer = 'glorot_uniform',
                     activation = 'relu'
                    )
              ) 
classifier.add(Dense(units = 1150,
                     kernel_initializer = 'glorot_uniform',
                     activation = 'relu'
                    )
              )    
classifier.add(Dense(units = 292,
                     kernel_initializer = 'glorot_uniform',
                     activation = 'relu'
                    )
              )
classifier.add(Dense(units = 77,
                     kernel_initializer = 'glorot_uniform',
                     activation = 'relu'
                    )
              )
classifier.add(Dense(units = 23,
                     kernel_initializer = 'glorot_uniform',
                     activation = 'relu'
                    )
              )
classifier.add(Dense(units = 7,
                     kernel_initializer = 'glorot_uniform',
                     activation = 'relu'
                    )
              )
classifier.add(Dense(units = 4,
                     kernel_initializer = 'glorot_uniform',
                     activation = 'softmax'
                    )
              )

classifier.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

The above is my network. After training the network, I predict values and convert them to class labels using:

from sklearn.preprocessing import LabelBinarizer

labels = np.argmax(predictions, axis = -1)
lb = LabelBinarizer()
labeled_predictions = lb.fit_transform(labels)

Upon calling a classification report comparing y_test and labeled_predctions , I receive the error.

**As a side note for anyone curious, I am experimenting with natural language processing and neural networks. The reason the input vector of my network is so large is that it takes in count-vectorized text as part of its inputs.

---

Edit #2:

I converted the predictions into a dataframe and dropped duplicates for both the test set and predictions getting this result:

y_test.drop_duplicates()

      javascript  python    r   sql
 738           0       0    0     0
4678           1       0    0     0
6666           0       0    0     1
5089           0       1    0     0
6472           0       0    1     0

predictions_df.drop_duplicates()


     javascript python  r   sql
738           1      0  0     0
6666          0      0  0     1
5089          0      1  0     0
3444          0      0  1     0

So, essentially what's happening is due to the way softmax is being converted to binary, the predictions will never result in a [0,0,0,0] . When one hot encoding y_test , should I just not drop the first column?

Answer 1

Yes I would say that you should not drop the first column. Because what you do now is to get the softmax and then take the neuron with the highest value as label (labels = np.argmax(predictions, axis = -1) ). With this approach you can never get a [0,0,0,0] result vector. So instead of doing this just create a onehot vector with positions for all 5 classes. You're problem with sklearn should then disappear, as you will get samples with true labels for your 5th class.

I'm also not sure if the dummy variable trap is a problem for neural networks. I have never heard from this before and a short google scholar search did not find any results. Also in all resources I've seen so far about neural networks I never saw this problem. So I guess (but this is really just a guess), that it isn't really a problem that you have when training neural networks. This conclusion is also driven by the fact that the majority of NNs use a softmax at the end.