不平衡数据集的训练/验证/测试集

Question

I am working in an NLP task for a classification problem. My dataset is imbalanced and some authors have 1 only text, thus I want to have this text only in the training test. As for the other authors I have to have a spliting of 70%, 15% and 15% respectivelly.

I tried to use train_test_split function from sklearn, but the results aren't good.

My dataset is a dataframe and it looks like this

Title Preprocessed_Text Label

---

Please let me know.

Answer 1

Whit only One sample of a particular class it seems impossible to measure the classification performance on this class. So I recommend using one or more oversampling approaches to overcome the imbalance problem ([a hands-on article on it][1]). As a matter of fact, you must pay more attention to splitting the data in such a way that preserves the prior probability of each class (for example by setting the stratify argument in train_test_split ). In addition, there are some considerations about the scoring method you must take into account (for example accuracy is not the best fit for scoring).

Answer 2

It is rather hard to obtain good classification results for a class that contains only 1 instance (at least for that specific class). Regardless, for imbalanced datasets, one should use stratified train_test_split, which preserves the same proportions of instances in each class as observed in the original dataset.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.25)

I should also add that if the dataset is rather small, let's say no more than 100 instances, it would be preferable to use cross-validation instead of train_test_split, and more specifically, StratifiedKFold that returns stratified folds.

When it comes to evaluation, you should consider using metrics such as Precision, Recall and F1-score (the harmonic mean of the Precision and Recall), however, using the average weighted score for each of these, which uses a weight that depends on the number of true instances of each class. As per the documentation

'weighted':

Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance; it can result in an F-score that is not between precision and recall.