tf.keras的tf.data.Dataset性能

Question

I am playing with multilabel text classification using tf.keras (tensorflow version 1.9.0). I have a dataset consisting of 185485 train and 46372 validation examples.

For the first try (on CPU), I pre-padded the data and fed it to the model:

from tensorflow import keras
....
X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=2000)

inp = keras.Input(shape=(X_train.shape[1], ))
x = keras.layers.Embedding(len(tk.word_index) + 1, 256, mask_zero=True)(inp)
x = keras.layers.LSTM(128, return_sequences=False)(x)
x = keras.layers.Dropout(0.1)(x)
x = keras.layers.Dense(y_train.shape[1], activation="sigmoid")(x)
model = keras.Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=128, epochs=300, validation_split=0.2)

The model takes ~130 minutes per epoch to train.

I then try to do the same with Dataset. Since my data is large, I can't fit in 2GB limit to use Dataset.from_tensor_slices() , and I use Dataset.from_generator() instead

X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=2000)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=777, stratify=y_train)

def gen(data, labels):
    for x, y in zip(data, labels):
        yield x, y

train_dataset = tf.data.Dataset.from_generator(
    lambda: gen(X_train, y_train),
    output_types=(tf.int32, tf.int32),
    output_shapes=([2000], [y_train.shape[1]]),
)

train_dataset = train_dataset.batch(128)
train_dataset = train_dataset.repeat()

val_dataset = ...

....
model.fit(train_dataset, epochs=300, steps_per_epoch=len(X_train)//128, validation_data=val_dataset,
      validation_steps=len(X_val)//128)

I expected the performance to be roughly the same, which is not the case, since one epoch took ~280 minutes to train. What am I missing? How do I achieve the same performance using the Dataset input?

Answer 1

• Try unroll=True in LSTM layer. For detail check LSTM layer documentation on Keras website
• Or use CuDNNLSTM on GPU for further speedup