AWS Sagemaker Monitoring Error: Encoding mismatch

Question

I am trying to run model monitoring on a model in AWS Sagemaker. The monitoring jobs are failing due to " Encoding mismatch: Encoding is JSON for endpointInput, but Encoding is base64 for endpointOutput. We currently only support the same type of input and output encoding at the moment."

Encoding is JSON for endpointInput and base64 for endpointOutput but the expected is json for both input and output.

I tried using the json_content_types in the DataCaptureConfig but the endpointOutput is still in base64 encoded.

Below is my DataCaptureConfig which i used in the deploy:

data_capture_config=DataCaptureConfig(
enable_capture = True,
sampling_percentage=100,
json_content_types = 'application/json',
destination_s3_uri=MY_BUCKET)

My capture files from the model looks something like this:

{
"captureData": {
"endpointInput": {
"observedContentType": "application/json",
"mode": "INPUT",
"data": "{ === json data ===}",
"encoding": "JSON"
},
"endpointOutput": {
"observedContentType": "*/*",
"mode": "OUTPUT",
"data": "{====base 64 encoded output ===}",
"encoding": "BASE64"
}
},
"eventMetadata": {
=== some metadata ===
}

I have observed that the output content type is not being recognized as the json/application. So I need a workaround/procedure to get the output in the json encoded form. Please help to the get JSON encoding for both input and output data.

Similar issue reported here , but there is no response.

Answer 1

I have come across a similar issue earlier while invoking the endpoint using boto3 sagemaker-runtime. Try adding the 'Accept' request parameter in invoke_endpoint function with value as 'application/json'.

refer for more help https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html#API_runtime_InvokeEndpoint_RequestSyntax

Answer 2

While deploying the endpoint please set the CaptureContentTypeHeader in the DataCaptureConfig and appropriately map the output either Json ontentTypes or CsvContentTypes.

https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CaptureContentTypeHeader.html

Doing this would set the encoding accordingly. If this is not set the default is base64 encoding and hence the issue.