在 SageMaker 上更改训练模型中的预处理

Question

I have trained model on SageMaker together with prerocessing. By preprocessing I mean I added the inference.py file with input_handler and output_handler functions according to this https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/deploying_tensorflow_serving.rst .

I works nice but the problem is that everytime I want to change something in the preprocessing I have to retrain the model. Is there maybe some other to to do this without retraining?

Answer 1

A trained model is simply a function that gets arguments (the input vector) and returns an output (the output vector/value). If you changed the input with your modified pre-processing, you need to change the implementation of your function. This means that you need to retrain your model.

Retraining your models is a good habit, even if you don't change anything in your pre-processing, as the input is changing in time. The classical example of house prices is highlighting that your model is only good for the data that you trained on. If after a couple of years the market has changed, you have to retrained your model.

Some models are being retrained every day. Amazon SageMaker makes it easy to train your model, but calling the train API, and waiting for it to finish. You can automate the process of building a new Docker image (if you changed your pre-processing), calling the training API, and then calling the deployment API to SageMaker to ECS/EKS or any other container hosting service.