Can someone explain this pytorch neural network code ? Are there two different neural networks here or one?

Question

class doubleNetwork(nn.Module):

def __init__(self, input_dim, output_dim):
    super(doubleNetwork, self).__init__()
    self.policy1 = nn.Linear(input_dim, 256) 
    self.policy2 = nn.Linear(256, output_dim)

    self.value1 = nn.Linear(input_dim, 256)
    self.value2 = nn.Linear(256, 1)

def forward(self, state):
    logits = F.relu(self.policy1(state))
    logits = self.policy2(logits)

    value = F.relu(self.value1(state))
    value = self.value2(value)

    return logits, value

1. Are policy1 , value1 in different networks or the same?

2. Are there two different neural networks here or single?

3. What is happening in the code here?

Answer 1

You have two networks in parallel. You can see it in the forward method:

state -> policy1 -> policy2 -> logits

state -> value1 -> value2 -> value

policy1 , policy2 , value1 and value2 are 4 different and independent fully connected (Linear) layers. The nn.Linear method creates a new layer of neurons every time it's called.

Edit for more details:

In your code you define a doubleNetwork class, the __init__ method will be called when you create an object of this class

So this line:

my_network = doubleNetwork(10,15)

call the __init__ method, and create a new doubleNetwork object. The newtork will have 4 attributes value1, value1, policy1, policy2 that are fully connected layers.

the line:

self.policy1 = nn.Linear(input_dim, 256) 

Create a new Linear Object which is a fully connected layer, when this line is executed the weights for the layers are initialized.

The forward method of network defines what append when the network object is called. For example with a line like that:

output1, output2 = my_network(input)

The code written in the forward is the function applied to the input. Here the input which is state is passed in the two policy layers in one side and then passed into the two value layers. And then both outputs are returned. So the network in form of a fork with one input and 2 outputs.

In this code it's one network, but because the two outputs depend only of the input and are independent with each other we could have define them in two separate network with the same result. See the code for example:

class SingleNetwork(nn.Module):

def __init__(self, input_dim, output_dim):
    super(doubleNetwork, self).__init__()
    self.layer1 = nn.Linear(input_dim, 256) 
    self.layer2 = nn.Linear(256, output_dim)

def forward(self, state):
    output = F.relu(self.layer1(state))
    output = self.layer2(output)

    return output

my_network1 = singleNetwork(10,15)
my_network2 = singleNetwork(10,1)

then:

output1 = my_network1(input)
output2 = my_network2(input)

Will be equivalent to

 output1, output2 = my_network(input)