评估多个机器学习模型的正确方法

Question

I am writing a QARegression module in Python for my machine learning project where I want to evaluate multiple models. Suppose, this is a image recognition model running on multiple images located in multiple folders.

 - folder-1
   - img-1
   - img-2
   - img-3
 - folder-2
   - img-1
   ......

Does it matter if I write like this

for eachFolder in FolderList:
    for eachImage in ImageList:
        for eachModel in ModelList:
            evaluate(predicted, GroundTruth)

Or

for eachModel in ModelList:
    for eachFolder in FolderList:
        for eachImage in ImageList:
            evaluate(predicted, GroundTruth)

Where in the end I would want output like this

model_1 : score1
model_2 : score2
.
.
.

which will be better in terms of

1. runtime complexity

2. correctness

OR it does not matter at all, we can write in anyway?

Answer 1

It presumably depends on the externals. If each model takes a long time to load, you want to do it as infrequently as possible, so want the model loop outside. If each image is very large, you want to load those as infrequently as possible, so want the image loop to be outside the model loop.

Answer 2

Both of them will evaluate each models. But you should go for the second one. The second one will take one model and evaluate that model with all images in all folders. It is better than loading the models again and again for different images. Also it will be harder to evaluate all models with first code snippet.