Forward Propagation for Neural Network

Question

I am trying to create a forward-propagation function in Python 3.8.2. The inputs look like this:

Test_Training_Input = [(1,2,3,4),(1.45,16,5,4),(3,7,19,67)]
Test_Training_Output = [1,1,0]

I am not using biases (not sure if they are that important and it makes my code very complicated) but I am using weights. The weights are stored in a list, Layer1W , I'm not sure how long to make it, but I think, len(Test_Training_Input)+len(Test_Training_Output) should work.

So far, my function looks like this:

def forwardprop():
    global Layer1O
    Layer1O = []
    for init in range(0,len(Layer1W)):
        total = sum(Test_Training_Input[1][1])*Layer1W[init]
        Layer1O.append(relu(total))
    return Layer1O

I think this is very wrong... Any suggestions?

Answer 1

Look into using numpy, a library that works with matrices. Something like forward-propagation can be easily implemented like:

import numpy as np

for layer in layers:
  inputs = np.dot(inputs, layer)

# this returns the outputs after propogating

This is just a bare-bones example and I'm excluding a bunch of things like caching the inputs at each layer during propogation.

Answer 2

So generally it's not a good idea to initialize any variables in that function. Initialize all of the variables, maybe in a constructor before the function. Also I suggest manually creating or mapping the structure of the neural network before writing it. It might help to visualise the things you need to do before anything.

If you want to try and do forward propagation you need to break down the steps, what does it do? First you identify which layer you are on, if you are on the inputs then times the inputs matrix by the next weights, etc. This should make it easier to write your network as well as reduce the amount of mistakes you might make.

Answer 3

This is a very rough explanation of a considerably complex topic, I highly recommend this e-book

import numpy as np
import numpy.random as rnd

#inputs
x = np.array([[0, 1, 0],
             [1, 0, 1]])

# x is of shape 2 x 3

#Here is a 3 -> 2 network, layer 1 (L1) will have 3 neurons, layer 2 (L2) will have 2
#   n00 is the first neuron of layer 1
#   n01 is the second neuron of layer 2
#   n02 is the third neuron of layer 3

#   n10 is the first neuron of layer 2
#   n11 is the second neuron of layer 2

#so there will be 6 weights, (let us assume their values also)
#   W(n00->n10) = 0.1, W(n00->n11) = 0.2
#   W(n01->n10) = 0.3, W(n01->n11) = 0.4
#   W(n02->n10) = 0.5, W(n02->n11) = 0.6

# we will store the corresponding weights in this fashion
# you might wonder why make it 2 x 3 matrix, and not 3 x 2
# this is to make use of matrix multiplication

#                 n00   n01  n02
w =    np.array([[0.1, 0.3, 0.5],  #n10
                 [0.2, 0.4, 0.6]]) #n11

# so the weight between n02 and n11 = w[1, 2] = 0.6

# note the input should be a vector,
# for example if the input is like this x = [1,1,0], shape = 3
# we will feed it as a vector, i.e
# [[1], <- x0
#  [1], <- x1
#  [0]] <- x2
# shape =  3 x 1

# so there will be 2 outputs from the feedforward, given by
#o1 = w00*x0 + w01*x1 + w02*x2
#o2 = w10*x0 + w11*x1 + w12*x2

# this is just the matrix multiplication between w and x,
#
#   [[0.1, 0.3, 0.5],   X  [[1],
#    [0.2, 0.4, 0.6]]      [1],     =    [[o1],
#                          [0]]           [o2]]
#

def feedforward(w, x):
    z = np.matmul(w, x)
    return z

# we will make x a vector, i.e change the shape as 2 x (3 x 1)
x = x[:, :, np.newaxis]
print(x.shape)
print(feedforward(w, x[0]))


# for your example
x = np.array([[1,2,3,4],[1.45,16,5,4],[3,7,19,67]])
# make it to 3 x 4 x 1
x = x[:, :, np.newaxis]

#lets try a 4 -> 2 -> 1 network

w1 = rnd.normal(size=(2, 4))
w2 = rnd.normal(size=(1, 2))

wgt = [w1, w2]

def feedforward2(wght, x):
    # best practice would be to put this in a class and store the outputs of each layer
    for w in wght:
        x = np.matmul(w, x)
    return x 

for i in x:
    print(feedforward2(wgt, i))