Keras 修剪：将权重设置为零不会加速推理？

Question

I'm writing a pruning algorithm for tf.keras that simply removes the lowest x percentile of weights from a layer / filter. To do this, I've tried setting the value of the weights to prune to zero. Having read other sources, I'm under the impression that this has the same effect as "removing" a weight from a network, but even if I set all the weights in a network to be zero, no decrease in inference time is noted.

If I were to hypothetically set all the weights in a layer to zero, the code would be as follows:

    flat_weights = np.array(self.model.layers[layer_index].get_weights()[0]).flatten()

    weight_index = 0 
    for weight in flat_weights:
        #if weight < self.delta_percentiles[index]:
        flat_weights[weight_index] = 0
        weight_index += 1

    weights[0] = np.reshape(flat_weights, original_shape)
    weights[1] = np.zeros(np.shape(weights[1]))

    self.model.layers[index].set_weights(weights)

Theoretically, the inference time of a model pruned in such a way should decrease but no change is found. Am I pruning correctly?

Answer 1

Setting a weight to zero is kind of the same as removing a weight, as then the network would be functionally equivalent if you had the same architecture, but with the same weights and one less neuron in that layer. The predictions you would get are the same.

But it does not have an effect on computational performance, as you noticed. For computation time to change, you would have to define a new network with one less weight, and then load the weights from the other architecture. You are now imagining that doing this is not easy, and it is the reason why we do not do it generally for evaluation, as we want to find out how predictive performance (like accuracy or mean squared error) changes as you prune weights.

So in order to get the computational advantages of pruning, you have to do a lot more than just setting weights to zero.