How to get accurate predictions from a neural network

Question

I created below neural network for the truth table for the 3-input logic AND gate, but the expected output for the [1,1,0] is not correct.Output should be 0. But it predicts as 0.9 that means approximately 1. So the output is not correct. So what I need to know is how to make the output prediction more accurate.Please guide me.

import numpy as np

class NeuralNetwork():
    def __init__(self):

        self.X = np.array([[0, 0, 0],
                          [0, 0, 1],
                          [0, 1, 0],
                          [0, 1, 1],
                          [1, 0, 0],
                          [1, 0, 1],
                          [1, 1, 1]])

        self.y = np.array([[0],
                           [0],
                           [0],
                           [0],
                           [0],
                           [0],
                           [1]])

        np.random.seed(1)

        # randomly initialize our weights with mean 0
        self.syn0 = 2 * np.random.random((3, 4)) - 1
        self.syn1 = 2 * np.random.random((4, 1)) - 1

    def nonlin(self,x, deriv=False):
        if (deriv == True):
            return x * (1 - x)

        return 1 / (1 + np.exp(-x))

    def train(self,steps):
        for j in xrange(steps):

            # Feed forward through layers 0, 1, and 2
            l0 = self.X
            l1 = self.nonlin(np.dot(l0, self.syn0))
            l2 = self.nonlin(np.dot(l1, self.syn1))

            # how much did we miss the target value?
            l2_error = self.y - l2

            if (j % 10000) == 0:
                print "Error:" + str(np.mean(np.abs(l2_error)))

            # in what direction is the target value?
            # were we really sure? if so, don't change too much.
            l2_delta = l2_error * self.nonlin(l2, deriv=True)

            # how much did each l1 value contribute to the l2 error (according to the weights)?
            l1_error = l2_delta.dot(self.syn1.T)

            # in what direction is the target l1?
            # were we really sure? if so, don't change too much.
            l1_delta = l1_error * self.nonlin(l1, deriv=True)

            self.syn1 += l1.T.dot(l2_delta)
            self.syn0 += l0.T.dot(l1_delta)

        print("Output after training:")
        print(l2)

    def predict(self,newInput):
        # Multiply the input with weights and find its sigmoid activation for all layers
        layer0 = newInput
        print("predict -> layer 0 : "+str(layer0))
        layer1 = self.nonlin(np.dot(layer0, self.syn0))
        print("predict -> layer 1 : "+str(layer1))
        layer2 = self.nonlin(np.dot(layer1, self.syn1))
        print("predicted output is : "+str(layer2))




if __name__ == '__main__':
    ann=NeuralNetwork()
    ann.train(100000)
    ann.predict([1,1,0])

Output:

Error:0.48402933124
Error:0.00603525276229
Error:0.00407346660344
Error:0.00325224335386
Error:0.00277628698655
Error:0.00245737222701
Error:0.00222508289674
Error:0.00204641406194
Error:0.00190360175536
Error:0.00178613765229
Output after training:
[[  1.36893057e-04]
 [  5.80758383e-05]
 [  1.19857670e-03]
 [  1.85443483e-03]
 [  2.13949603e-03]
 [  2.19360982e-03]
 [  9.95769492e-01]]
predict -> layer 0 : [1, 1, 0]
predict -> layer 1 : [ 0.00998162  0.91479567  0.00690524  0.05241988]
predicted output is : [ 0.99515547]

Answer 1

Actually, it does produce correct output -- the model is ambiguous. Your input data fits A*B ; the value of the third input never affects the given output, so your model has no way to know that it's supposed to matter in case 110. In terms of pure information theory, you don't have the input to force the result you want.

Answer 2

Seems like this is happening for every input you miss in the AND gate. For example try replacing [0, 1, 1] input with [1, 1, 0] and then try to predict [0, 1, 1] it predicts the final value close to 1 . I tried including biases and learning rate but nothing seem to work.

Like Prune mentioned it might be because the BackPropagation Network is not able to work with the incomplete model.

To train your network to the fullest and get optimal weights, provide all the possible inputs ie 8 inputs to the AND gate. Then you can always get the correct predictions because you already trained the network with those inputs, which might not make sense with predictions in this case. May be predictions on a small dataset do not work that great.

This is just my guess because almost all the networks I used for predictions used to have fairly bigger datasets.