如何使用 tensorflow 或 keras 重新训练线性回归 model 和新子集？

Question

I have 100 Gb of data and divided it into small subsets. I want to train the model in an incremental way using a new subset until all the algorithm is trained on all the subsets. How I can achieve this TensorFlow or sklearn?

Answer 1

Some scikit-learn models do support incremental learning through the partial_fit method. A popular choice is the Stochastic Gradient Descent, which minimizes a loss function looking at one data sample at a time. Here is an example, assuming you have two chunks of data that you can load successively to memory, (X1, y1), (X2, y2) .

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
sgd = SGDRegressor(random_state=42)

X1_scaled = scaler.partial_fit(X1).transform(X1)
sgd.partial_fit(X1_scaled, y1)

X2_scaled = scaler.partial_fit(X2).transform(X2)
sgd.partial_fit(X2_scaled, y2)