改变神经网络中 Sigmoid 激活的阈值

Question

Hi I am new to machine learning and I have a query about changing thresholds for sigmoid function.

I know Sigmoid function's value is in the range [0;1], 0.5 is taken as a threshold, if h(theta) < 0.5 we assume that it's value is 0, if h(theta) >= 0.5 then it's 1.

Thresholds are used only on the output layer of the network and it's only when classifying. So, if you're trying to classify between 3 classes can you give different thresholds for each class (0.2,0.4,0.4 - for each class)? Or can you specify a different threshold overall, like 0.8? I am unsure how to define this in the code below. Any guidance is appreciated.

# Hyper Parameters
input_size = 14
hidden_size = 40
hidden_size2 = 30
num_classes = 3
num_epochs = 600
batch_size = 34
learning_rate = 0.01


class Net(torch.nn.Module):
    def __init__(self, n_input, n_hidden, n_hidden2, n_output):
        super(Net, self).__init__()
        # define linear hidden layer output
        self.hidden = torch.nn.Linear(n_input, n_hidden)
        self.hidden2 = torch.nn.Linear(n_hidden, n_hidden2)
        # define linear output layer output
        self.out = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        """
            In the forward function we define the process of performing
            forward pass, that is to accept a Variable of input
            data, x, and return a Variable of output data, y_pred.
        """
        # get hidden layer input
        h_input1 = self.hidden(x)
        # define activation function for hidden layer
        h_output1 = torch.sigmoid(h_input1)

        # get hidden layer input
        h_input2 = self.hidden2(h_output1)
        # define activation function for hidden layer
        h_output2 = torch.sigmoid(h_input2)

        # get output layer output
        out = self.out(h_output2)

        return out


net = Net(input_size, hidden_size, hidden_size, num_classes)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

all_losses = []

for epoch in range(num_epochs):
    total = 0
    correct = 0
    total_loss = 0
    for step, (batch_x, batch_y) in enumerate(train_loader):
        X = batch_x
        Y = batch_y.long()

    # Forward + Backward + Optimize
    optimizer.zero_grad()  # zero the gradient buffer
    outputs = net(X)
    loss = criterion(outputs, Y)
    all_losses.append(loss.item())
    loss.backward()
    optimizer.step()

    if epoch % 50 == 0:
        _, predicted = torch.max(outputs, 1)
        # calculate and print accuracy
        total = total + predicted.size(0)
        correct = correct + sum(predicted.data.numpy() == Y.data.numpy())
        total_loss = total_loss + loss
    if epoch % 50 == 0:
        print(
            "Epoch [%d/%d], Loss: %.4f, Accuracy: %.2f %%"
            % (epoch + 1, num_epochs, total_loss, 100 * correct / total)
        )

train_input = train_data.iloc[:, :input_size]
train_target = train_data.iloc[:, input_size]

inputs = torch.Tensor(train_input.values).float()
targets = torch.Tensor(train_target.values - 1).long()

outputs = net(inputs)
_, predicted = torch.max(outputs, 1)

Answer 1

You can use any threshold you find suitable.

Neural networks are known to be often over-confident (eg applying 0.95 to one of 50 classes), so it may be beneficial to use different threshold in your case.

Your training is fine, but you should change predictions (last two lines) and use torch.nn.softmax like this:

outputs = net(inputs) 
probabilities = torch.nn.functional.softmax(outputs, 1)

As mentioned in other answer you will get each row with probabilities summing to 1 (previously you had unnormalized probabilities aka logits).

Now, just use your desired threshold on those probabilities:

predictions = probabilities > 0.8

Please notice you may get only zeros in some cases (eg [0.2, 0.3, 0.5] ).

This would mean neural network isn't confident enough according to your standards and would probably drop number of incorrect positive predictions (abstract, but say you are predicting whether a patient doesn't have one of mutually exclusive 3 diseases. It's better to say so only if you are really sure).

Different thresholds for each class

This could be done as well like this:

thresholds = torch.tensor([0.1, 0.1, 0.8]).unsqueeze(0)
predictions = probabilities > thresholds

Final comments

Please notice in case of softmax only one class should be the answer (as pointed out in another answer) and this approach (and mention of sigmoid) may indicate you are after multilabel classification .

If you want to train your network so it can simultaneously predict classes you should use sigmoid and change your loss to torch.nn.BCEWithLogitsLoss .

Answer 2

In a multi class classification you should have an output for each class. Then you can use the softmax function to normalize the output, so the sum of all of them is 1. The output with the biggest value is the one chosen as the classification.