My tensorflow neural network accuracy does not change

Question

I wants to build a neural network for Student Admission dataset (admit, gre, gpa, rank) I made admit and rank one-hot as follows

    one_hot_data = pd.concat([data, pd.get_dummies(data['rank'], prefix='rank')], axis=1)
    one_hot_data = pd.concat([one_hot_data, pd.get_dummies(data['admit'], prefix='admit')], axis=1)
    
    # Drop the previous rank column
    data = one_hot_data.drop('rank', axis=1)
    data = one_hot_data.drop('admit', axis=1)
    print(data.shape)

I split the data using train_test_split and scale using minmax_scale But neural network is as folows

n_features = X_train.shape[1]
n_labels = y_train.shape[1]

features = tf.placeholder(tf.float32, [None, n_features])
labels = tf.placeholder(tf.float32, [None, n_labels])

w = [
    tf.Variable(tf.random_normal((n_features, 16)), name='Weights_layer_0'),
    tf.Variable(tf.random_normal((16, 4)), name='Weights_layer_1'),
    tf.Variable(tf.random_normal((4, n_labels)), name='Weights_layer_2'),
]

n_layers = len(w)
b = [
    tf.Variable(tf.zeros(16), name='Bias_layer_0'),
    tf.Variable(tf.zeros(4), name='Bias_layer_1'),
    tf.Variable(tf.zeros(n_labels), name='Bias_layer_2'),
]
def neural_network(input, weights, biases):
    for i in range(n_layers-1):
        layer = tf.add(tf.matmul(input if i==0 else layer, weights[i]),biases[i])
        layer = tf.nn.relu(layer)
        # layer = tf.nn.dropout(layer, keep_prob=0.6)
    out_layer = tf.add(tf.matmul(layer, weights[-1]),biases[-1])
    return out_layer

loss_ = []
res = []
prediction = neural_network(features, w, b)
loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=prediction, labels=labels))
optim = tf.train.AdadeltaOptimizer(0.0001).minimize(loss)
correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
with tf.device('/gpu'):
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        for i in range(10):
            for m,n in zip(X_train_batches, y_train_batches):
                _, l = sess.run([optim, loss],feed_dict={features: m, labels: n})
            loss_.append(l)

            acc = sess.run([accuracy], feed_dict={features: X_train, labels: y_train})
            print(i, acc)
        test_accuracy = sess.run(accuracy,feed_dict={features: X_test, labels: y_test})
        print(test_accuracy)
        res = sess.run(neural_network(features,w,b),feed_dict={features: X})

But accuracy doesn't change

0 [0.4857143]
1 [0.4857143]
2 [0.4857143]
3 [0.4857143]
4 [0.4857143]
5 [0.4857143]
6 [0.4857143]
7 [0.4857143]
8 [0.4857143]
9 [0.4857143]
10 [0.4857143]
0.5333333

and loss stays the same

[0.5546836, 0.5546756, 0.5546678, 0.55466014, 0.55465263, 0.5546452, 0.55463773, 0.55463034, 0.5546232, 0.5546159, 0.5546088, 0.5546016, 0.5545944, 0.5545874, 0.5545803, 0.5545734, 0.55456626, 0.5545592, 0.5545522, 0.5545452]

What is missing? Is my neural network correct? Full code

Answer 1

There may be many possible causes here (and we don't have your data), but, according to my experience, a frequent mistake in such cases is initializing the weights with the default argument of stddev=1.0 in tf.random_normal() (see the docs ), as you do here.

A stddev=1.0 is a huge value, and it alone can make your NN go astray. Change it to stddev=0.01 for all your initial weights:

w = [
    tf.Variable(tf.random_normal((n_features, 16), stddev=0.01), name='Weights_layer_0'),
    tf.Variable(tf.random_normal((16, 4), stddev=0.01), name='Weights_layer_1'),
    tf.Variable(tf.random_normal((4, n_labels), stddev=0.01), name='Weights_layer_2'),
    ]

Other than that, as already suggested in the comments, a learning rate of 0.0001 seems way too small here (given how slowly the loss is decreasing); experiment with higher values ( 0.01 - 0.001 ).