平衡数据集后的低 F1 分数

Question

I have a binary classification problem with tweets; 17000 as a positive class and 122000 as a negative class. I have balanced the data with both as 17000 tweets in each class. I have implemented models like LR, SVM, BERT, LSTM and CNN. In every run, the F1 score is around 0.55-0.66. Am I doing something wrong? Is it normal to have an F1 score around 0.55?

The problem continues with another dataset also. The sample BERT model is

trainer = Trainer(
model=model,                      # our loaded pre-trained transformer-based model "DistilBERT"
args=training_args,               # our defined training arguments
train_dataset=train_dataset,      # training dataset
eval_dataset=eval_dataset,        # evaluation dataset
compute_metrics=compute_metrics   # our defined evaluation function 

)

Answer 1

Althought an F1 score around 0.55-0.66 can be normal it depends on your use case if it is good enough.

The F1 score depends not only on data balancing but also on lots of factors.

I'd recommend to take into account this steps for your ML pipeline:

1. Data preparation/cleansing (tokenization, stop-words removal, etc.)

2. Algorithms/Model selection (from experience SVM and NN perform good)

3. Feature engineering/selection (Which features have more influence in the model)

4. Hyperparameter tunning (depending on the model you'll have to search for the better combination of hyperparameters. Eg with NN you have to specify how many layers, how many nodes, activation fn, back propagation fn, etc.)

Many people like to focus on the last steps but I'd say that data preparation is one of the most important steps in any data pipeline. Data preparation/cleansing plays also a huge role in the F1 score and practicaly all other metrics.