在一张图像中多次出现相同的 label - Tensorflow

Question

I'm trying to create a tf model, which can detect any handwriting in any image. In order to do that, i made the labels in all train pictures with just one label: edit. It means, one image can have this labels many times.

After many hours of training using cpu i did't get the expected result. The model can't see any of the blocks i gave before training.

I'm using the following model:

http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d6_coco17_tpu-32.tar.gz

Is the problem that i'm labeling one image with one label multiple times?

Could be the problem of using cpu instead of gpu? I have currently one gpu with 4gb and it seems not enough. i trained the model with 2000 steps and learning_rate was 0.006. should i train it to be more than that?

Any suggestions?

Thank you in advanced.

Edit

Following the is a screenshot from tensorboard of the trained model:

Answer 1

CPU vs GPU

GPU has the only advantage that the training is faster. It shouldn't have any effect on the expected result. It just takes longer. Though, for some models, the difference could be large. Monitoring your training might give you more insight.

Training monitoring

What does it mean you did not get the expected result? How was it different from what you expected?

I suggest you use some kind of monitoring, such as Tensorboard to monitor both the loss and metrics of the training and validation dataset if you do not already do so. This will give you invaluable information about the training in real-time.

Pipeline debugging

When your model seems not to be learning anything, you must start debugging. You can follow the following steps to make sure that none of those is your problem: https://blog.slavv.com/37-reasons-why-your-neural.network-is-not-working-4020854bd607

I especially like to overfit the model on a single batch. This test tells me whether my algorithmic pipeline is correct, including preprocessing, the model, and the evaluation. The result should be that evaluation on the single training batch should give you a very good score, while for the rest of the data the score will be poor since the model will be overfitted on the single batch.

Problem definition

Sometimes, it is possible that the problem is wrongly defined. Can you clarify what your labels are? I do not fully understand. If it is a common problem, then you can search the inte.net to see the inspiration for how it is usually defined.

EDIT

Loss functions

In general, you want all your loss functions to go down. In your case, they go up in the first few hundred steps, which might not necessarily indicate that there's something wrong because sometimes it takes a short time in the beginning before the training stabilizes. N.netheless, the triangles indicate that the loss was NaN. That means that there is something wrong. I can recommend using the tf.keras.callbacks.TerminateOnNaN callback to detect the NaNs in your loss immediately, which will terminate your training promptly.

Metrics

From the loss functions themselves, it hard to tell what is the model's performance. In every machine learning task, you have to be able to understand the performance of a model. Metrics are used exactly for that purpose. In this case (object detection), I suggest using IoU to determine how well the predicted box overlaps with the target one, and precision and recall for evaluating the performance of your binary classification of whether the predicted box contains hand-written text or not.