为什么我不能用这个网络和约束学习 XOR 函数？

Question

Let's say I have the following constraints and the network:

1. The architecture is fixed ( see this image ) (note that there are no biases)
2. Activation function for the hidden layer is ReLU
3. There's no activation function for the output layer (should just return the sum of the inputs it receive).

I tried to implement this in pytorch with various initialization schemes and different data sets but I failed (the code is at the bottom).

My questions are:

1. Is there anything wrong with my NN training process?
2. Is this a feasible problem? If yes, how?
3. If this is doable, can we still achieve that by constraining the weights to be in the set {-1, 0, 1}

---

Code:

import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data as data_utils
import numpy as np

class Network(nn.Module):

     def __init__(self):

        super(Network, self).__init__()
        self.fc1 = nn.Linear(2,2,bias=False)
        self.fc2 = nn.Linear(2,1, bias=False)
        self.rl = nn.ReLU()

      def forward(self, x):
        x = self.fc1(x)
        x = self.rl(x)
        x = self.fc2(x)
        return x 

#create an XOR data set to train    
rng = np.random.RandomState(0)
X = rng.randn(200, 2)
y = np.logical_xor(X[:, 0] > 0, X[:, 1] > 0).astype('int32')

# test data set
X_test = np.array([[0,0],[0,1], [1,0], [1,1]])

train = data_utils.TensorDataset(torch.from_numpy(X).float(), \
                         torch.from_numpy(y).float())
train_loader = data_utils.DataLoader(train, batch_size=50, shuffle=True)

test = torch.from_numpy(X_test).float()

# training the network
num_epoch = 10000

net = Network()
net.fc1.weight.data.clamp_(min=-1, max=1)
net.fc2.weight.data.clamp_(min=-1, max=1)

# define loss and optimizer
criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters())

for epoch in range(num_epoch):
  running_loss = 0 # loss per epoch
  for (X, y)in train_loader:
    # make the grads zero
    optimizer.zero_grad()
    # forward propagate
    out = net(X)
    # calculate loss and update
    loss = criterion(out, y)
    loss.backward()
    optimizer.step()
    running_loss += loss.data
  if epoch%500== 0:      
      print("Epoch: {0} Loss: {1}".format(epoch, running_loss))

The loss doesn't improve. It gets stuck in some value after a few epochs ( i'm not sure how to make this reproducible as I'm getting different values every time)

net(test) returns a set of predictions that are no way close to XOR output.

Answer 1

You need to use a nonlinear activation function such as sigmoid in your hidden and output layers . because xor is not linearly separable.Also biases are required.