TF model 精度在保存和加载后改变

Question

I trained a tf model for classification, and the auc on validate data was 0.759 after 1 epoch.

I saved the checkpoint, then restored and continue training with same train data. The validate auc suddenly jumped to 0.764, and converged to 0.766 before over-fitting.

And the auc is also 0.766 on test data set.

I wonder how it happened. Why TF model accuracy changed after saving and loading?

---

I also try to train 2 epochs at a time, without Save/Load. After the first epoch, the model began to over fit rapidly.

Answer 1

I wonder how it happened. Why TF model accuracy changed after saving and loading?

The accuracy increased because you trained it for one more epoch. Loading and saving has nothing to do with it. This is to be expected. And the vice versa is also true. Your accuracy might also decrease due to overfitting. The number of epochs to train your network is also a hyperparameter that we need to tune. Early stopping can help in this problem. Also, you get a different set of weights everytime you compile a model. You can fix the random seed to get more reproducible results.

Because of the non deterministic nature of neural networks( meaning you won't always get the same output for the same input, because with random weight initialization you might end up in different minimas), it is standard practice to reset and run your network several times(perhaps 10 times) and take the min, max and stddev of the performance metric. And low stddev suggests that you have a stable architecture. You can expect it to behave similarly everytime you train it.