Gradients of Logical Operators in Tensorflow

Question

I'm trying to create a very simple binary classifier in Tensorflow on generated data.

I'm generating random data from two separate normal distributions. Then I will classify the resulting data to a binary class if it it less than or greater than a number, A.

Ideally, A will be a cutoff in the middle of both normals. Eg if my data is generated by N(1,1) + N(-1,1), then A should be approximately 0.

I'm runnning into a "No gradients provided for any variable..." error. Specifically:

No gradients provided for any variable: ((None, <tensorflow.python.ops.variables.Variable object at 0x7fd9e3fae710>),)

I think it may have to do with the fact that Tensorflow cannot calculate gradients for logical operators. My classification for any given A value is supposed to be something like:

Given a data point x and an A value:

[1,0] : if x < A

[0,1] : if x >= A

Given that idea, here is my calculation in Tensorflow for the output:

my_output = tf.concat(0,[tf.to_float(tf.less(x_data, A)), tf.to_float(tf.greater_equal(x_data, A))])

Is this the wrong way to implement this output? Is there a non-logical functional equivalent?

Thanks. If you want to see my whole code, here is a gist: https://gist.github.com/nfmcclure/46c323f0a55ae1628808f7a58b5d437f

---

Edit : Full Stack Trace:

Traceback (most recent call last):

  File "<ipython-input-182-f8837927493d>", line 1, in <module>
    runfile('/.../back_propagation.py', wdir='/')

  File "/usr/local/lib/python3.4/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 699, in runfile
execfile(filename, namespace)

  File "/usr/local/lib/python3.4/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 88, in execfile
exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

  File "/.../back_propagation.py", line 94, in <module>
train_step = my_opt.minimize(xentropy)

  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/optimizer.py", line 192, in minimize
name=name)

  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/optimizer.py", line 286, in apply_gradients
(grads_and_vars,))

ValueError: No gradients provided for any variable: ((None, <tensorflow.python.ops.variables.Variable object at 0x7fd9e3fae710>),)

Answer 1

Typically you would use a sigmoid function to pin the output of your function to the range of 0 to 1. You want to train the following function:

y = a*x_input + b, where a and b are trainable variables.

The loss function you would use would then be tf.sigmoid_cross_entropy_with_logits

And to evaluate the class you would evaluate sigmoid(y) > 0.5. The greater than logical operator does not have a gradient to create an optimization function.

Answer 2

The thresholding function you're trying to use isn't differentiable at the threshold, and the derivative is zero everywhere else. As szabadaba mentioned, you'd typically use a sigmoidal function as the output of a binary classifier. For multiclass classification, use a softmax function. These functions can be interpreted as giving you a probability or confidence for the class value (which is nicer than just getting a hard value). Importantly, the gradients are well behaved. But, you have to take care about not saturating the units.

For example, see:

LeCun (1998). Efficient BackProp.

In the case of binary classification w/ sigmoidal outputs, he makes some interesting points about staying in the non-saturated regime by choosing the particular sigmoid function, the right mapping to class labels for that sigmoid, etc.

It would probably be helpful to read a logistic regression tutorial. Here's one for TensorFlow and one for Theano .

Answer 3

我认为也许您不应该使用my_output计算损失，因为也许tf无法支持tf.less和tf.greater操作中的向后梯度，您可以尝试计算x_data中的损失函数

Answer 4

Just change my_output as follow

my_output = tf.concat(0,[tf.to_float(tf.less(tf.div(x_data,A), 1.)), 
tf.to_float(tf.greater_equal(tf.div(x_data, A),1.0)])