张量数组的归一化

Question

Experts. I am new to DNN and Python. I am trying to use tensorflow to do some DNN learning work. During my working, I came across a problem that I myself cannot solve. In one step, I would like to normalize a tensor called "inputs". The normalization is simply take the maximum abs of a vector, and divide all the elements of the vector my the maximum abs. But the following problem occured:

ValueError Traceback (most recent call last) in ()

 55             tmp_index = tf.argmax(tmp_abs,0)
 56             tmp_index1 = tf.cast(tmp_index,dtype = tf.int32)
---> 57             inputs = inputs/tmp_abs[tmp_index1]
 58 
 59         if index != len(Layers)-1:

InvalidArgumentError: Shape must be rank 1 but is rank 2 for 'hidden2_3/strided_slice' (op: 'StridedSlice') with input shapes: [?,1], [1,1], [1,1], [1].

Any advice will be appreciated. Thanks!

# input features and labels
x_ = tf.placeholder(name="input", shape=[None, 1], dtype=np.float32)
y_ = tf.placeholder(name="output", shape=[None, 1], dtype=np.float32)

# tf variables
Hidden = []

# Hidden Layers
for index, num_hidden in enumerate(Layers):
    with tf.name_scope("hidden{}".format(index+1)):
        if index == 0:
            weights = tf.Variable(tf.truncated_normal([Fea_Size,num_hidden], stddev = get_stddev(Fea_Size,num_hidden)))
            bias = tf.Variable(tf.zeros([num_hidden]))
        else:
            weights = tf.Variable(tf.truncated_normal([Layers[index-1], num_hidden], stddev = get_stddev(Layers[index-1], num_hidden)))
            bias = tf.Variable(tf.zeros([num_hidden]))

        inputs = x_ if index == 0 else Hidden[index-1]
        if index !=0:
            tmp_abs = tf.abs(inputs)
            tmp_index = tf.argmax(tmp_abs,0)
            tmp_index1 = tf.cast(tmp_index,dtype = tf.int32)
            inputs = inputs/tmp_abs[tmp_index1]

        if index != len(Layers)-1:
            Hidden.append(tf.nn.relu(tf.matmul(inputs,weights) + bias))
        else:
            nonlin_model = tf.nn.relu(tf.matmul(inputs,weights) + bias)

nonlin_loss = tf.reduce_mean(tf.pow(nonlin_model - y_, 2), name='cost')
train_step_nonlin = tf.train.GradientDescentOptimizer(0.01).minimize(nonlin_loss)

Answer 1

The problem is with the fact that your inputs has invariant shape for axis 0 .

You can instead use:

inputs = inputs/tf.reduce_max(tf.abs(inputs))

tf.abs returns the absolute value of inputs . tf.reduce_max returns the maximum value.

Here's a code snippet that worked for me:

inputs = tf.placeholder(shape=[None, 1], dtype = tf.float32)
inputs_normal = inputs/tf.reduce_max(tf.abs(inputs))

all_pos = sess.run(inputs_normal, feed_dict={inputs:[[1],[2],[3]]})
all_neg = sess.run(inputs_normal, feed_dict={inputs:[[-1],[-2],[-3]]})
pos_neg = sess.run(inputs_normal, feed_dict={inputs:[[1],[2],[-3]]})

Here are the values:

all_pos = array([[0.33333334],
                 [0.6666667 ],
                 [1.        ]], dtype=float32)

all_neg = array([[-0.33333334],
                 [-0.6666667 ],
                 [-1.        ]], dtype=float32)

pos_neg = array([[ 0.33333334],
                      [ 0.6666667 ],
                      [-1.        ]], dtype=float32)

I've shown it for 2-D tensors, but it should work for higher dimensional tensors as well.