带有少量数据集的浅层神经网络的训练速度

Question

My data contains of 1 feature and a label per feature ie. ["smallBigTest", "toastBob"] <- feature 4 labels ["mix", "small", "big", "medium"]

I have converted my features to numbers based on alphabet ie.

smallBigTest -> 18, 12,  0, 53, 53, 27,  8,  6, 45,  4, 18, 19
toastBob ->     19, 14,  0, 18, 19, 27, 14,  1, -1, -1, -1, -1

which later on I hot-encoded and reshaped so the final array of features would look like [[hotencoded(18,12,0,53,53,27,8,6,45,4,18,19)], [hotencoded(19,14,0,18,19,27,14,1,-1,-1,-1,-1)]

simply made it into a 2d array from 3d array to match my labels shape, i have also hot encoded labels

the training data is about 60k lines of text 1.2mb csv file

and here is my model:

model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(16, activation=tf.nn.sigmoid))
model.add(tf.keras.layers.Dense(labelsDictSize, activation=tf.nn.softmax))

optimizer = tf.train.GradientDescentOptimizer(0.05)
model.compile(optimizer, loss=tf.losses.softmax_cross_entropy)
model.fit(featuresOneHot,labelsOneHot, steps_per_epoch=dataCount, epochs=5, verbose=1)

I'm new to ML, so I might be doing something completely wrong or completely stupid, I thought though that this amount of data would be fine. Training on my machine with gtx870m takes an hour per epoch and on google collaboratory around 20-30 minutes per epoch

Answer 1

It's not unusual for NLP models to take so much time to train. The only thing I would change about your model to speed up the learning process is changing the optimizer to something that doesn't have a fixed learning rate. That should speed up the process. I would suggest using adam as it's one of the fastest optimizers with good performance.

Just replace

model.compile(optimizer, loss=tf.losses.softmax_cross_entropy)

with

model.compile(optimizer='adam', loss=tf.losses.softmax_cross_entropy)