使用單個神經網絡 model 訓練多個數據集

Question

我有的：

1. 一個神經網絡model
2. 10 個相同的結構化數據集

我想要的是：

1. 分別訓練所有數據集
2. 分別保存他們的模型

我可以分別訓練數據集並一次保存一個模型。 但我想在一次運行中加載我的 10 個數據集並使用它們創建 10 個模型。 解決方案可能很明顯，但我對此很陌生。 我如何實現這一目標？

提前致謝。

Answer 1

您可以使用concurrency and parallelism的概念之一，即Multi-Threading ，或者在某些情況下， Multi-Processing來實現這一點。
最簡單的編碼方法是使用 python 的concurrent-futures模塊。

對於要使用的每個數據集，您可以在 model 上調用訓練 function，所有這些都在 ThreadPoolExecutor 下，以便觸發並行線程以執行單獨的訓練。

代碼可能有點像這樣：

from concurrent.futures import ThreadPoolExecutor, as_completed

import tensorflow as tf
from tensorflow.keras.models import load_model, Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten



def create_model():                                                 # responsible for creating model
    model = Sequential()
    model.add(Flatten())                                            # adding NN layers
    model.add(Dense(64))
    model.add(Activation('relu'))
    # ........ so on
    model.compile(optimizer='..', loss='..', metrics=[...])         # compiling the model
    return model                                                    # finally returning the model


def load_fit(model, XY_train):                                      # performs model.fit(...parameters...)
    model.fit(XY_train[0], XY_train[1], epochs=5, validation_split=0.3)     # use your already defined x_train, y_train
    return model                                                    # finally returns trained model



# trains provided model on each dataset parallelly by using multi-threading
def parallel_trainer(model, XY_train_datasets : list[tuple]):
    with ThreadPoolExecutor(max_workers = len(XY_train_datasets)) as executor:
        futureObjs = [
            executor.submit(
                lambda ds: load_fit(model, ds), *XY_train_datasets) # Call Load_Fit for each dataset iterrate through the datasets
            ]

        for i, obj in enumerate(as_completed(futureObjs)):          # iterate through trained models
            (obj.result()).save(f"{i}.model")                       # save models




model = create_model()                                              # create the model

mnist = tf.keras.datasets.mnist                                     # get dataset - for example :- mnist dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()            # get (x_train, y_train), (x_test, y_test)
datasets = [(x_train, y_train)]*10                                  # list of dataset paths (in your case, same dataset used 10 times)

parallel_trainer(model, datasets)                                   # call parallel trainer