Slow training of BERT model Hugging face

Question

I am training the binary classfier using BERT model implement in hugging face library

training_args = TrainingArguments(
   "deleted_tweets_trainer",                  
   num_train_epochs = 1,            
   #logging_steps=100,    
   evaluation_strategy='steps',       
   remove_unused_columns = True    
)

I am using Colab TPU still the training time is a lot, 38 hours for 60 hours cleaned tweets.

Is there any way to optimise the training?

Answer 1

You are currently evaluating every 500 steps and have a training and eval batch size of 8.

Depending on your current memory consumption, you can increase the batch sizes (eval much more as training consumes more memory):

• per_device_train_batch_size
• per_device_eval_batch_size

In case it matches your use case, you can also increase the steps after an evaluation is started;

• eval_steps