Implement a Regression/Classification Neural Network in Python

Question

Problem Context

I am trying to learn Neural Networks in Python, & I've developed an implementation of a Logistic Regression based NN.

Here is the code -

import numpy as np

# Input array
X = np.array([[1, 0, 1, 0], [1, 0, 1, 1], [0, 1, 0, 1]])

# Output
y = np.array([[1], [1], [0]])


# Sigmoid Function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))


# Derivative of Sigmoid Function
def ddx_sigmoid(x):
    return x * (1 - x)


#####   Initialization - BEGIN  #####


# Setting training iterations
iterations_max = 500000

# Learning Rate
alpha = 0.5

# Number of Neruons in Input Layer = Number of Features in the data set
inputlayer_neurons = X.shape[1]

# Number of Neurons in the Hidden Layer
hiddenlayer_neurons = 3  # number of hidden layers neurons

# Number of Neurons at the Output Layer
output_neurons = 1  # number of neurons at output layer

# weight and bias initialization
wh = np.random.uniform(size=(inputlayer_neurons, hiddenlayer_neurons))
bh = np.random.uniform(size=(1, hiddenlayer_neurons))
wout = np.random.uniform(size=(hiddenlayer_neurons, output_neurons))
bout = np.random.uniform(size=(1, output_neurons))

#####   Initialization - END  #####

# Printing of shapes

print "\nShape X: ", X.shape, "\nShape Y: ", y.shape
print "\nShape WH: ", wh.shape, "\nShape BH: ", bh.shape, "\nShape Wout: ", wout.shape, "\nShape Bout: ", bout.shape

# Printing of Values
print "\nwh:\n", wh, "\n\nbh: ", bh, "\n\nwout:\n", wout, "\n\nbout: ", bout

#####   TRAINING - BEGIN  #####
for i in range(iterations_max):
    #####   Forward Propagation - BEGIN   #####

    # Input to Hidden Layer = (Dot Product of Input Layer and Weights) + Bias
    hidden_layer_input = (np.dot(X, wh)) + bh

    # Activation of input to Hidden Layer by using Sigmoid Function
    hiddenlayer_activations = sigmoid(hidden_layer_input)

    # Input to Output Layer = (Dot Product of Hidden Layer Activations and Weights) + Bias
    output_layer_input = np.dot(hiddenlayer_activations, wout) + bout

    # Activation of input to Output Layer by using Sigmoid Function
    output = sigmoid(output_layer_input)

    #####   Forward Propagation - END #####

    #####   Backward Propagation - BEGIN   #####

    E = y - output

    slope_output_layer = ddx_sigmoid(output)
    slope_hidden_layer = ddx_sigmoid(hiddenlayer_activations)

    d_output = E * slope_output_layer

    Error_at_hidden_layer = d_output.dot(wout.T)
    d_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer

    wout += hiddenlayer_activations.T.dot(d_output) * alpha
    bout += np.sum(d_output, axis=0, keepdims=True) * alpha

    wh += X.T.dot(d_hiddenlayer) * alpha
    bh += np.sum(d_hiddenlayer, axis=0, keepdims=True) * alpha

    #####   Backward Propagation - END   #####
#####   TRAINING - END  #####

print "\nOutput is:\n", output

This code is working perfectly in the cases where the output is binary (0,1). I guess, this is because of the sigmoid function I am using.

Problem

Now, I want to scale this code so that it can handle linear regression as well.

As we know, the scikit library has some preloaded datasets which can be used for classification and regression.

I want my NN to train and test the diabetes dataset.

With this in mind, I have modified my code as follows -

import numpy as np
from sklearn import datasets

# Sigmoid Function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))


# Derivative of Sigmoid Function
def ddx_sigmoid(x):
    return x * (1 - x)

# Load Data
def load_data():
    diabetes_data = datasets.load_diabetes()
    return diabetes_data

input_data = load_data()

X = input_data.data

# Reshape Output
y = input_data.target
y = y.reshape(len(y), 1)

iterations_max = 1000

# Learning Rate
alpha = 0.5

# Number of Neruons in Input Layer = Number of Features in the data set
inputlayer_neurons = X.shape[1]

# Number of Neurons in the Hidden Layer
hiddenlayer_neurons = 5  # number of hidden layers neurons

# Number of Neurons at the Output Layer
output_neurons = 3  # number of neurons at output layer

# weight and bias initialization
wh = np.random.uniform(size=(inputlayer_neurons, hiddenlayer_neurons))
bh = np.random.uniform(size=(1, hiddenlayer_neurons))
wout = np.random.uniform(size=(hiddenlayer_neurons, output_neurons))
bout = np.random.uniform(size=(1, output_neurons))


#####   TRAINING - BEGIN  #####
for i in range(iterations_max):
    #####   Forward Propagation - BEGIN   #####

    # Input to Hidden Layer = (Dot Product of Input Layer and Weights) + Bias
    hidden_layer_input = (np.dot(X, wh)) + bh

    # Activation of input to Hidden Layer by using Sigmoid Function
    hiddenlayer_activations = sigmoid(hidden_layer_input)

    # Input to Output Layer = (Dot Product of Hidden Layer Activations and Weights) + Bias
    output_layer_input = np.dot(hiddenlayer_activations, wout) + bout

    # Activation of input to Output Layer by using Sigmoid Function
    output = sigmoid(output_layer_input)

    #####   Forward Propagation - END #####

    #####   Backward Propagation - BEGIN   #####

    E = y - output

    slope_output_layer = ddx_sigmoid(output)
    slope_hidden_layer = ddx_sigmoid(hiddenlayer_activations)

    d_output = E * slope_output_layer

    Error_at_hidden_layer = d_output.dot(wout.T)
    d_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer

    wout += hiddenlayer_activations.T.dot(d_output) * alpha
    bout += np.sum(d_output, axis=0, keepdims=True) * alpha

    wh += X.T.dot(d_hiddenlayer) * alpha
    bh += np.sum(d_hiddenlayer, axis=0, keepdims=True) * alpha

    #####   Backward Propagation - END   #####
#####   TRAINING - END  #####

print "\nOutput is:\n", output

The output of this code is -

Output is:
[[ 1.  1.  1.]
 [ 1.  1.  1.]
 [ 1.  1.  1.]
 ..., 
 [ 1.  1.  1.]
 [ 1.  1.  1.]
 [ 1.  1.  1.]]

Obviously, I am goofing up somewhere in the basics.

Is this because I am using the sigmoid function for the hidden as well as the output layer?

What kind of functions shall I use so that I get a valid output, which can be used to train my NN efficiently?

Efforts so far

I have tried to use the TANH function, the SOFTPLUS function as activation to both the layers, without any success.

Can someone out here please help?

I tried Googling this but the explanations there are very complex.

HELP !

Answer 1

You should try to remove sigmoid function on your output.

As for linear regression, output range may be large, while output of sigmoid or tanh function is [0, 1] or [-1, 1], which makes minimize of error function not possible.

======= UPDATE ======

I try to accomplish it in tensorflow, the core part for it:

w = tf.Variable(tf.truncated_normal([features, FLAGS.hidden_unit], stddev=0.35))
b = tf.Variable(tf.zeros([FLAGS.hidden_unit]))

# right way
y = tf.reduce_sum(tf.matmul(x, w) + b, 1)

# wrong way: as sigmoid output in [-1, 1]
# y = tf.sigmoid(tf.matmul(x, w) + b)

mse_loss = tf.reduce_sum(tf.pow(y - y_, 2) / 2