XOR神经网络2-1-1

Question

I am trying to implement a XOR in neural networks with the typology of 2 inputs, 1 element in the hidden layer, and 1 output. But the learning rate is really bad (0,5). I think it is because I am missing a connection between the inputs AND the outputs, but I am not really sure how to do it. I have already made the bias connection so that the learning is better. Only using Numpy.

def sigmoid_output_to_derivative(output):
  return output*(1-output)
a=0.1
X = np.array([[0,0],
[0,1],
[1,0],
[1,1]])
np.random.seed(1)

y = np.array([[0],
[1],
[1],
[0]])
bias = np.ones(4)
X = np.c_[bias, X]

synapse_0 = 2*np.random.random((3,1)) - 1
synapse_1 = 2*np.random.random((1,1)) - 1

for j in (0,600000):

  layer_0 = X
  layer_1 = sigmoid(np.dot(layer_0,synapse_0))
  layer_2 = sigmoid(np.dot(layer_1,synapse_1))
  layer_2_error = layer_2 - y

  if (j% 10000) == 0:
    print( "Error after "+str(j)+" iterations:" + str(np.mean(np.abs(layer_2_error))))

  layer_2_delta = layer_2_error*sigmoid_output_to_derivative(layer_2)

  layer_1_error = layer_2_delta.dot(synapse_1.T)

  layer_1_delta = layer_1_error * sigmoid_output_to_derivative(layer_1)

  synapse_1 -= a *(layer_1.T.dot(layer_2_delta))
  synapse_0 -= a *(layer_0.T.dot(layer_1_delta))

Answer 1

You need to be careful with statements like

the learning rate is bad

Usually the learning rate is the step size that gradient descent takes in negative gradient direction. So, I'm not sure what you mean by a bad learning rate.

I'm also not sure if I understand your code correctly, but the forward step of a neural net is basically a matrix multiplication of the weight matrix for the hidden layer times the input vector. This will (if you set up everything correctly) result in a matrix which is equal to the size of your hidden layer. Now, you can simply add the bias before applying your logistic function elementwise to this matrix.

h_i = f(h_i+bias_in)

Afterwards you can do the same thing for the hidden layer times the output weights and apply its activation to get the outputs.

o_j = f(o_j+bias_h)

The backwards step is to calculate the deltas at output and hidden layer including another elementwise operation with your function

sigmoid_output_to_derivative(output)

and update both weight matrices using the gradients (here the learning rate is needed to define the step size). The gradients are simply the value of a corresponding node times its delta. Note: The deltas are differently calculated for output and hidden nodes.

I'd advice you to keep separate variables for the biases. Because modern approaches usually update those by summing up the deltas of its connected notes times a different learning rate and subtract this product from the specific bias.

Take a look at the following tutorial (it uses numpy):

http://peterroelants.github.io/posts/neural_network_implementation_part04/