Keras: multi class imbalanced data classification is overfitting

Question

I have a small dataset of ~1000 rows with two categorical columns [Message] , [Intent] . I want to create a classification model and make predictions for new, unseen messages.

The 29 unique intents are imbalanced, ranging from 116 to 4 value counts:

intent_1                     116
intent_2                     98
intent_3                     81
intent_4                     78
intent_5                     73
intent_6                     68
intent_7                     66
intent_8                     65
intent_9                     62
intent_10                    61
intent_11                    56
intent_12                    53
intent_13                    50
intent_14                    49
intent_15                    45
intent_16                    40
intent_17                    37
intent_18                    32
intent_19                    31
intent_20                    30
intent_21                    25
intent_22                    22
intent_23                    21
intent_24                    19
intent_25                    15
intent_26                    12
intent_27                    10
intent_28                    9
intent_29                    4

I first tried a CNN model and metrics accuracy and categorical cross-entropy, but the results were not good:

After adjusting parameters such as batch size, epochs, drop-off, activation function, pre-trained word embeddings, and also trying a bi-lstm model, my model was still over-fitting.

So, I tried to add an over-sampling step before I train the model, in order to tackle class imbalance, but now the results are even worse.

Can anyone help me? Here's my code so far:

#load training and validation files and concatenate them

def load_datasets(train_file, val_file):
  train = pd.read_excel(train_file)
  val = pd.read_csv(val_file, error_bad_lines=False, sep=";")
  #drop additional info column
  train.drop(train.columns[1], axis=1, inplace=True)
  #concatenate two dataframes
  frames = [train, val]
  result = pd.concat(frames)  

  intent = result["Label"]
  unique_intent = list(set(intent))
  sentences = list(result["Message"])

  return(intent, unique_intent, sentences)

intent, unique_intent, sentences = load_datasets("Training.xlsx","Validation.csv")

#define stemmer 
stemmer = SnowballStemmer("english")

#define lemmatizer
lemmatizer = WordNetLemmatizer() 

#clean the data : remove punctuation, tokenize, lowercase, lemmatize
def preprocessing(sentences):
  words = []
  for sent in sentences:
    clean = re.sub(r'[^ a-z A-Z 0-9]', " ", sent)
    w = word_tokenize(clean)
    #stemming
    words.append([lemmatizer.lemmatize(i.lower()) for i in w])

  return words

cleaned = preprocessing(sentences)


#creating tokenizer
def create_tokenizer(words, filters = '!"#$%&()*+,-./:;<=>?@[\]^_`{|}~'):
  token = Tokenizer(filters = filters)
  token.fit_on_texts(words)

  return token

#defining maximum length
def max_length(words):

  return(len(max(words, key = len)))


#show vocabulary size and maximum length 
word_tokenizer = create_tokenizer(cleaned)
vocab_size = len(word_tokenizer.word_index) + 1
max_length = max_length(cleaned)

print("Vocab Size = %d and Maximum length = %d" % (vocab_size, max_length))

#### Vocab Size = 811 and Maximum length = 45

#encoding list of words
def encoding_doc(token, words):

  return(token.texts_to_sequences(words))

encoded_doc = encoding_doc(word_tokenizer, cleaned)


#add padding to make words of equal length to use in the model
def padding_doc(encoded_doc, max_length):

  return(pad_sequences(encoded_doc, maxlen = max_length, 
                       padding =   "post"))



padded_doc = padding_doc(encoded_doc, max_length)



output_tokenizer = create_tokenizer(unique_intent,
                        filters = '!"#$%&()*+,-/:;<=>?@[\]^`{|}~')


output_tokenizer.word_index

encoded_output = encoding_doc(output_tokenizer, intent)

encoded_output = np.array(encoded_output).reshape(len(encoded_output), 1)


#one-hot encoding
def one_hot(encode):
  o = OneHotEncoder(sparse=False, categories='auto')

  return(o.fit_transform(encode))


output_one_hot = one_hot(encoded_output)


#split dataset to train (70%) and validation set (30%)
from sklearn.model_selection import train_test_split

train_X, val_X, train_Y, val_Y = train_test_split(padded_doc, output_one_hot, 
                                                   shuffle = True, 
                                                  random_state=1,
                                                  test_size = 0.3)

print("Shape of train_X = %s and train_Y = %s" % (train_X.shape, train_Y.shape))
print("Shape of val_X = %s and val_Y = %s" % (val_X.shape, val_Y.shape))

#### Shape of train_X = (929, 45) and train_Y = (929, 29)
#### Shape of val_X = (399, 45) and val_Y = (399, 29)


#Over Sample 

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import recall_score


clf_rf = RandomForestClassifier(n_estimators=10)
clf_rf.fit(x_train_res, y_train_res)


#over sampling
from sklearn.model_selection import train_test_split
from imblearn.over_sampling import SMOTENC

x_train, x_val, y_train, y_val = train_test_split(train_X, train_Y, 
                                                   shuffle = True, 
                                                  test_size = 0.3)


sm = SMOTENC(categorical_features=[0,44], k_neighbors=2)

x_train_res, y_train_res = sm.fit_sample(x_train, y_train)


print('Validation Results')
print(clf_rf.score(x_val, y_val))
print(recall_score(y_val, clf_rf.predict(x_val),average='micro'))
print('\nTest Results')
print(clf_rf.score(val_X, val_Y))
print(recall_score(val_Y, clf_rf.predict(val_X),average='micro'))


### Validation Results
### 0.4495798319327731
### 0.4495798319327731

### Test Results
### 0.14035087719298245
### 0.14035087719298245

#Define the model

def create_model(vocab_size, max_length):
  model = Sequential()
  model.add(Embedding(vocab_size, 100, input_length=max_length, trainable=False))
  model.add(Dropout(0.2))
  model.add(Conv1D(64, 5, activation='relu'))
  model.add(MaxPooling1D(pool_size=4))
  model.add(LSTM(32))
  model.add(Dense(29, activation='softmax'))

  return model


model = create_model(vocab_size, max_length)

model.compile(loss = "categorical_crossentropy", optimizer = "adam", 
              metrics = ["accuracy"])
model.summary()


filename = 'model.h5'
checkpoint = ModelCheckpoint(filename, monitor='val_loss', verbose=1, 
                             save_best_only=True, mode='min')

hist = model.fit(x_train_res, y_train_res, epochs = 100, batch_size = 32, 
                 validation_data = (val_X, val_Y), callbacks = [checkpoint])


model = load_model("model.h5")


plt.style.use('ggplot')

def plot_history(hist):
  acc = hist.history['acc']
  val_acc = hist.history['val_acc']
  loss = hist.history['loss']
  val_loss = hist.history['val_loss']
  x = range(1, len(acc)+1)

  plt.figure(figsize=(12,5))
  plt.subplot(1, 2, 1)
  plt.plot(x, acc, 'b', label='Training acc')
  plt.plot(x, val_acc, 'r', label='Validation acc')
  plt.title('Training and validation accuracy')
  plt.legend()
  plt.subplot(1,2,2)
  plt.plot(x, loss, 'b', label='Training loss')
  plt.plot(x, val_loss, 'r', label='Validation loss')
  plt.title('Training and validation loss')
  plt.legend()


plot_history(hist)

Answer 1

A few observations:

1. You can observe that the first model that you have (CNN + cross-entropy) was already overfitting. Switching towards a more complex model does not help at all in case of overfitting; on the contrary, you get worse results , as you can see in your second plot.
2. You have very little data . Even 1000 rows for 2 perfectly balanced classes(500 intent 1, 500 intent 2) is very few for a neural network, let alone 29 intents, which are unbalanced, and yet only 1000 rows.

Suggestions:

1. Create a very basic model. Do not use Bidirectional LSTM with many units, but a simple CNN with a few layers.
2. Use oversampling to balance your dataset(several techniques like SMOTE(I see you already use this) or ADASYN).
3. Create a more complex dataset, with many more rows, suitable for a neural network(gather more real data, no oversampling). Otherwise, I would switch to a classical machine learning model.
4. If the problem allows, data augmentation is also a solution(comment below @Joseph Budin).