为什么Tensorflow中二元分类头的logit_dimension=1？

Question

I am having a hard time understanding how the binary classification head works in Tensorflow. I am trying to create a custom multi-head Estimator in Tensorflow. My code looks like the following:

def model_fn_multihead(features, labels, mode, params):
    # Create simple heads and specify head name.
    head_target_0 = tf.contrib.estimator.binary_classification_head(name=target_0)

    head_target_1 = tf.contrib.estimator.multi_class_head(n_classes=3, name=target_1)

    # Create multi-head from two simple heads.
    head = tf.contrib.estimator.multi_head([head_target_0, head_target_1])
    # Create logits for each head, and combine them into a dict.

    net = tf.feature_column.input_layer(features, params['feature_columns'])
    for idx, units in enumerate(params['hidden_units']):
        net = tf.layers.dense(net, units=units, activation=tf.nn.relu, name='fully_connected_%d' % idx)

    # Compute logits (1 per class).
    logits_0 = tf.layers.dense(net, 2, activation=None, name='logits_0')
    logits_1 = tf.layers.dense(net, 3, activation=None, name='logits_1')

    logits = {target_0: logits_0, target_1: logits_1}

    def _train_op_fn(loss):
        return tf.train.AdagradOptimizer(learning_rate=0.01).minimize(loss, global_step=tf.train.get_global_step())    

    return head.create_estimator_spec(features=features, labels=labels, mode=mode, logits=logits, train_op_fn=_train_op_fn)

The problem is if I run the code as is, Tensorflow complains that logits_0 has the wrong dimensions, if I dig into the source code at tensorflow\\contrib\\estimator\\python\\estimator\\multi_head.py , it is expecting the logits dimension of "1' for "logits_0", but the clearly in a binary classifier there are two classes. What's going on? If I set the dimension to "1", the code will run but I will always get non-sensical results in the training. Basically the classifier can't learn the difference between a 1/0 target even with a single trivial feature.

This code works perfectly for multiple, multi-class heads (n_class>2).

I am using Tensorflow 1.4. Am I simply misunderstanding something? Perhaps my input is formatted incorrectly?

Update:

I figured out what the problem is, which is that Tensorflow is expecting a tensor of type "bool", it is not enough to submit labels of 1, 0, 0, 1, etc, wrapping the label with tf.equal(label, 1) solved the issue. Now I understand why the logits_dimension is 1. However, this still does not solve my actual problem. Which is that the binary classifier just doesn't seem to be working when wrapped in a multi_head. The classification results are just always wrong.

If we submit a simple trivial example involved a single categorical variable called: CAT_XXX where XXX is a number between 1 and 100. If we construct two target variables;

• Target_2: 0 if XXX%2==0, 1 if XXX%3==0, else 2
• Target_3: 0 if XXX%2==0, else 1

we can construct a trivial multi-headed, multi-classification problem. In such a scenario, I obtain results like:

accuracy/Target_2: 1.0
accuracy/Target_3: 0.600072
accuracy_baseline/Target_3: 0.600072
auc/Target_3: 0.497585
auc_precision_recall/Target_3: 0.399472
average_loss/Target_2: 0.000260735
average_loss/Target_3: 0.673509
global_step: 11720
label/mean/Target_3: 0.399928
loss: 21.5472
prediction/mean/Target_3: 0.399628

you can see the multi-class target has been perfectly predicted but the binary problem is nonsense. The thing is the binary_classification head works fine as a standalone input to DNNEstimator. It's just when it is wrapped in a multi_head things seem to go wrong.

Kuhan

Answer 1

The binary_classification_head() has the logit dimension of "1", because the probability of a binary classification is [alpha, 1-alpha] for the two classes.

According to the tensorflow binary_classification_head doc , if 'label_vocabulary' is not given, labels must be float Tensor with values in the interval [0, 1]. However, I tested the head with wrapping the labels to boolean (your suggestion) and casting the labels to integer or floats and they give same results. Probably tensorflow is doing an additional cast in the background.

Since the binary_classification_head() is working for the isolated case and not in the combination with multi_head(), I would recommend you to experiment with your hyperparameters (especially the learning rate). Have in mind that you change the calculation of the loss with the combination of heads and therefore you need to adjust your hyperparameters of your model. Maybe a good start would be to pass head_weights into the multi_head() with your old set of hyperparameters and see how your model reacts (just a thought).