Tuning Sagemaker Pipeline

Question

Is it possible to do a hyper-parameter optimization on Sagemaker for a flow (eg, pipeline) of a processing job followed by a training job?

In Sagemaker pipelines, I see I can use the tuner step with any training step. However, I can't see any helpful resource for integrating the processing job into the optimization.

Any ideas on how to do this task without merging two steps into one step?

In this relatively old question , it was asking about optimizing 2 models jointly. Here, I am asking about Processing and Training jobs.

I really appreciate any help you can provide.

Answer 1

There is no way to run a HyperparameterTuner for Processor, it expects an estimator as an input. On the other hand, you can just put your processing logic with hyperparameters into an appropriate Estimator ( sklearn for example) and then output processing script with hyperparameters as an artefact from /opt/ml/model/model.joblib into a S3 or as a model artefact. When done tuning, just load it into Processor via model_dir , and you got your Processor with tuned hyperparameters.