通过将权重转换为float16或int来减小模型尺寸

Question

I've a keras model for which I need to reduce its size. What I understood is that I can reduce the size by converting weights stored in layers to float16 or to int.

I did try to convert float16 and int with below code.

# Iterate over all the layers of the network
for layer_idx, layer in enumerate(model.layers):

    # If layer has no weights the move to next layer
    if not layer.get_weights():
        continue

    # Get existing weights
    old_weights = layer.get_weights()

    # List to store new weights
    new_weights = []

    # Iterate over weights
    for idx, weight in enumerate(old_weights):
        # Convert weight and append to new list
        new_weights.append(weight.astype(int))
        # print(weight.dtype)

    model.get_layer(name=layer.name).set_weights(new_weights)

For float16 the, the model size was not reduced and for int I converted weights using above code but facing below error while loading model.

 File "network_pruning.py", line 24, in <module>
    custom_objects={'angle_error': angle_error})
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/models.py", line 239, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/models.py", line 313, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 139, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/engine/topology.py", line 2490, in from_config
    process_layer(layer_data)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/engine/topology.py", line 2476, in process_layer
    custom_objects=custom_objects)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 141, in deserialize_keras_object
    return cls.from_config(config['config'])
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/engine/topology.py", line 1253, in from_config
    return cls(**config)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/engine/topology.py", line 1348, in __init__
    name=self.name)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 488, in placeholder
    x = tf.placeholder(dtype, shape=shape, name=name)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1777, in placeholder
    return gen_array_ops.placeholder(dtype=dtype, shape=shape, name=name)
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4516, in placeholder
    dtype = _execute.make_type(dtype, "dtype")
  File "/home/aditya/miniconda3/envs/python36/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 126, in make_type
    (arg_name, repr(v)))
TypeError: Expected DataType for argument 'dtype' not 'int'.

I am not even sure this is a right way of reducing model size or not. It will also be good you anyone can show me ways to reduce model size and complexity.

Many thanks in advance!!

Answer 1

I would not be using integers as weights. I can't imagine them being able to be granular enough for you to get the small weight changes needed to solvers like gradient descent etc. to eventually find the global minima(?).

Put another way, most learning rates values like 0.0001 or 0.001 which when eventually added to or subtracted from (with a bunch of math done to the learning rate) a weight will not change the weights value as anything to the right of the decimal will get dropped due the dtype being an int.