如何在ANN中设置偏置并将Sigmoid更改为ReLU功能？

Question

I'm trying to create a data prediction model through artificial neural networks. The following code is part of the Python-based ANN code created through many books. Also, the error rate between the predicted value and the actual value doesn't meet below 19%. I tried to increase the number of hidden layers, but it did not tremendously affect the error rate. I think this is probably a limitation of Sigmoid function and not considering Bias. I looked around for a month and found out how to build ReLU and Bias, but I could not find the range of Bias and ReLU.

Q1 = How do I convert Sigmoid to ReLU and Q2 = how to add Bias to my code?

Q3 = Also, If I change Sigmoid to ReLU, do I have to make my dataset 0.0~1.0 range? This is because Sigmoid function accepts 0.0~1.0 range of data, but I don't know what range ReLU allows.

I'm sorry to ask an elementary question.

class neuralNetwork:
# initialize the neural network
def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):

#
    self.inodes = input_nodes
    self.hnodes = hidden_nodes
    self.onodes = output_nodes

    # link weight matrices, wih and who
    self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
    self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))

    # learning rate
    self.lr = learning_rate

    # activation function is the sigmoid function
    self.activation_function = lambda x: scipy.special.expit(x)

    pass

# train the neural network
def train(self, inputs_list, targets_list):
    # convert inputs list to 2d array
    inputs = numpy.array(inputs_list, ndmin=2).T
    targets = numpy.array(targets_list, ndmin=2).T

    # calculate signals into hidden layer
    hidden_inputs = numpy.dot(self.wih, inputs)
    # calculate the signals emerging from hidden layer
    hidden_outputs = self.activation_function(hidden_inputs)

    # calculate signals into final output layer
    final_inputs = numpy.dot(self.who, hidden_outputs)
    # calculate the signals emerging from final output layer
    final_outputs = self.activation_function(final_inputs)

    # output layer error is the (target - actual)
    output_errors = targets - final_outputs
    # hidden layer error is the output_errors, split by weights, recombined at hidden nodes
    hidden_errors = numpy.dot(self.who.T, output_errors)

    # update the weights for the links between the hidden and output layers
    self.who += self.lr*numpy.dot((output_errors*final_outputs*(1.0-final_outputs)), numpy.transpose(hidden_outputs))
    # update the weights for the links between the input and output layers
    self.wih += self.lr*numpy.dot((hidden_errors*hidden_outputs*(1.0-hidden_outputs)), numpy.transpose(inputs))

    pass

# query the neural network
def query(self, inputs_list) :

    inputs = numpy.array(inputs_list, ndmin=2).T

    # convert hidden list to 2d array
    hidden_inputs = numpy.dot(self.wih, inputs)
    # calculate signals into hidden layer
    hidden_outputs = self.activation_function(hidden_inputs)


    final_inputs = numpy.dot(self.who, hidden_outputs)
    final_outputs = self.activation_function(final_inputs)
    return final_outputs        
    pass

Answer 1

Your question is too broad and there are lots of concept behind ReLU vs sigmoid.

But in short:
Sigmoid Saturate and kill gradients (look at Gradient descent ) sigmoid are not zero centered because output of sigmoid is 0<output<1 . I can see for sigmoid you are using scipy but for ReLU its easy. Relu is defined by the following function

f(x) = max(0,x)

This means if the input is greater then zero return input else return 0. And ReLU is prefered for hidden layers and other like softmax for output layers.

I would say, look different activation function and why we need activation functions on neural net. How sigmoid kills gradients and why they slow converge.

Q1 = How do I convert Sigmoid to ReLU and Q2 = how to add Bias to my code?
simply write a method on your own based on the ReLU function above and update the following line

self.activation_function = max(0,x) # instead of lambda x: scipy.special.expit(x)

Q3 = Also, If I change Sigmoid to ReLU, do I have to make my dataset 0.0~1.0 range? This is because Sigmoid function accepts 0.0~1.0 range of data, but I don't know what range ReLU allows.

Answer of this question depends on your network and your data, but yes you normalize the data. And there is no such range that you need to make your data in. Because for ReLU: If the input is less than zero, it will return 0 and if input is >= 0, it will return the input . So no such range like in sigmoid. Answer of this question

If you wanna look at how ReLU works and can be used, following detailed example will help though these examples are written using framework (PyTorch) to build the network and train.

• PyTorch Basic Projects Link
• ReLU vs sigmoid vs TanH Video