TF2.0：变压器 model ValueError：形状 (40759, 128) 和 (40765, 128) 不兼容

Question

The Transformer model has the following params. I saved and reloaded the model using h5py. I get this errors only for few datasets.

h5f = h5py.File(path + '.model.weights.h5', 'w')

# Weights reloaded

variables = []
    h5f = h5py.File(path + '.model.weights.h5', 'r')
    for idx in sorted([int(i) for i in h5f]):
        variables.append(np.array(h5f[str(idx)]))
    h5f.close()
    for idx, t in enumerate(this.model.trainable_variables):
        t.assign(variables[idx])

The hyperparameters to train the model are:

BUFFER_SIZE = 20000
BATCH_SIZE = 64
MAX_LENGTH = 40
num_layers = 4
d_model = 128
dff = 512
num_heads = 8
input_vocab_size = tokenizer_pt.vocab_size + 2
target_vocab_size = tokenizer_en.vocab_size + 2
dropout_rate = 0.1

transformer = Transformer(num_layers, d_model, num_heads, dff,
                          input_vocab_size, target_vocab_size, 
                          pe_input=input_vocab_size, 
                          pe_target=target_vocab_size,
                          rate=dropout_rate)

Once I reload the model, I get the following error. I could save all the parameters, but load fails with incompatibility error. What do those tensor shapes indicate?

raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (40759, 128) and (40765, 128) are incompatible

Traceback:

File "/Users/Models/Model.py", line 400, in load
    t.assign(modelTrainables[idx])
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 600, in assign
    self._shape.assert_is_compatible_with(value_tensor.shape)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/tensor_shape.py", line 700, in assert_is_compatible_with
    raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (40759, 128) and (40765, 128) are incompatible

Answer 1

The problem was with loading the tokenizers. There were punctuations which were getting saved while using the save function, but on load, some of these failed to load (don't know the root cause) and that caused the mismatch in shapes. Once I cleaned the text to remove punctuation and extra spaces, it worked fine.