How is bias added to specific hidden layers of a Neural Network in PyBrain?

Question

I have a Neural Network with two hidden layers. I want to add a bias unit only to the second hidden layer. How do I do that?

The code for my network is as follows:

nn = FeedForwardNetwork()
inLayer = LinearLayer(numFeatures)
hiddenLayer1 = LinearLayer(numFeatures+1)
hiddenLayer2 = SigmoidLayer(numFeatures+1)
outLayer = LinearLayer(1)

nn.addInputModule(inLayer)
nn.addModule(hiddenLayer1)
nn.addModule(hiddenLayer2)
nn.addOutputModule(outLayer)

in_to_hidden1 = FullConnection(inLayer, hiddenLayer1)
hidden1_to_hidden2 = FullConnection(hiddenLayer1, hiddenLayer2)
hidden2_to_out = FullConnection(hiddenLayer2, outLayer)

nn.addConnection(in_to_hidden1)
nn.addConnection(hidden1_to_hidden2)
nn.addConnection(hidden2_to_out)
nn.sortModules()

Answer 1

This is fairly simple task. First you have to create Bias module:

bias = BiasUnit()

Then add it to your NeuralNetwork, so:

nn = FeedForwadNetwork()
nn.addModule(bias)

Then, assuming you already added other layers you have to connect bias to hidden layer of your choice:

bias_to_hiden = FullConnection(bias, hiden_layer)

And then add it to neural network:

nn.addConnection(bias_to_hiden)

Apart from that you do everything same as before.

For the reference check code of the buildNetwork function from pybrain.tools.shortcuts module . Here is some code that connects bias unit to other layers (lines 75-79):

if opt['bias']:
    # add bias module and connection to out module, if desired
    n.addModule(BiasUnit(name='bias'))
    if opt['outputbias']:
        n.addConnection(FullConnection(n['bias'], n['out']))

Hope that helped.