Keras：将mask_zero与带填充序列相比使用单序列非带填充训练

Question

I'm building an LSTM model in Keras to classify entities from sentences. I'm experimenting with both zero padded sequences and the mask_zero parameter, or a generator to train the model on one sentence (or batches of same length sentences) at a time so I don't need to pad them with zeros.

If I define my model as such:

model = Sequential()
model.add(Embedding(input_dim=vocab_size+1, output_dim=200, mask_zero=True,
                    weights=[pretrained_weights], trainable = True))
model.add(Bidirectional(LSTM(units=100, return_sequences=True, recurrent_dropout=0.1)))
model.add(Dropout(0.2))
model.add(Bidirectional(LSTM(units=100, return_sequences=True, recurrent_dropout=0.1)))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(target_size, activation='softmax')))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics = ['accuracy'])

Can I expect the padded sequences with the mask_zero parameter to perform similarly to feeding the model non-padded sequences one sentence at a time? Essentially:

model.fit(padded_x, padded_y, batch_size=128, epochs=n_epochs,
 validation_split=0.1, verbose=1)

or

def iter_sentences():
        while True:
            for i in range(len(train_x)):
                yield np.array([train_x[i]]), to_categorical([train_y[i]], num_classes = target_size)

model.fit_generator(iter_sentences(), steps_per_epoch=less_steps, epochs=way_more_epochs, verbose=1)

I'm just not sure if there is a general preference for one method over the other, or the exact effect the mask_zero parameter has on the model.

Note: There are slight parameter differences for the model initialization based on which training method I'm using - I've left those out for brevity.

Answer 1

The biggest difference will be performance and training stability , otherwise padding and then masking is the same as processing single sentence at time.

1. performance : Well you will train one point at a time which might not exploit any parallelism that is available on the hardware. Often, we adjust the batch size to get the best performance from the machine during training and prediction.
2. training stability : when you set batch size to 1 you are not longer performing mini-batch training. The training routine will apply updates after every data point which might be detrimental for momentum based algorithms such as Adam. Instead, accumulating gradients over a batch tends to provide more stable convergence especially if the data is noisy.

So to answer the question, no, you can't expect them to perform similarly.