SageMaker: How to manually run a Model Quality Monitor Job?

Question

I'm using the SageMaker SDK class ModelQualityMonitor to configure Model Evaluation over time.

The monitoring schedule does not support monthly or weekly time periods, in my case, the ground Truth period is longer than one month.

There's an example from SageMaker Labs to run Data Quality Monitor manually, the idea seems to provide the input constraints and statistics with a ProcessingInput , but not sure how to do this.

Is it possible to run the SageMaker Model Quality Monitor job manually?

Answer 1

Model Monitoring out of the box is currently accomplished by launching SageMaker Processing Jobs on a schedule with data capture input.

In the case of ModelQualityMonitor , two Processing Jobs are launched running two different containers. Namely, sagemaker-model-monitor-groundtruth-merger and sagemaker-model-monitor-analyzer

Manual execution of ModelQualityMonitor is currently not supported out of the box. That being said, similar to the example you shared which showcases how to manually trigger the DataQualityMonitor Processing Job you could do the same for ModelQualityMonitor . For example you could look at launching the Processing Jobs and making sure the input/output is compliant. There is an extra level of complexity in the fact that there is the sagemaker-model-monitor-groundtruth-merger Processing Job that merges Data Capture to the Ground Truth data. The sagemaker-model-monitor-analyzer container then uses this merged data as input.

As such, the Data Capture format and folder structure, start and end times environment variables, etc. all need to be compliant.

To understand more on how ModelQualityMonitor is accomplished I suggest running this example and investigating the Processing Jobs that are launched and their input/output.