SageMaker Endpoint：调用 InvokeEndpoint 操作时 ServiceUnavailable 503

Question

I've deployed a model as SageMaker Endpoint, it worked fine for some time but now when invoking the model through boto3

import boto3

client = boto3.client('sagemaker-runtime')

response = client.invoke_endpoint(
        EndpointName="my-sagemaker-endpoint",
        ContentType="text/csv",
        Body=payload,
)

I got the following error

ServiceUnavailable: An error occurred (ServiceUnavailable) when calling the InvokeEndpoint operation (reached max retries: 4): A transient exception occurred while retrieving variant instances. Please try again later.

Researching about this error in SageMaker Documentation it states the following

The request has failed due to a temporary failure of the server.

I've also checked the Instance Metrics in CW and there's nothing unusual.

I'm not sure why this error is happening, any suggestions will be helpful.

Answer 1

TL; DR The error originates because the Instance is unable to retrieve the SageMaker Model artifact from s3.

Explanation

SageMaker Endpoints implement a /ping route which check if model artifact is able to load within the Instance. The model artifacts is first retrieved from s3 and then loaded into the instance. If model is not available on s3 it shows the following error (image below)

As the model artifact can't be retrieved from s3 because it was accidentally deleted, it can't be loaded which raises the error No such file or directory when calling the /ping route to check if the endpoint is healthy (see image below)

This in turn makes the Load Balancer to assume the instance has some problem, blocking you access to it, so when you try to invoke the endpoint you get a 503: Service Unavailable Error

Solution

I worked this out only by redeploying to a new endpoint but this time considering the following:

• At least num_instances=2 to guarantee each instance is at a different AZ, and the LB communicates with at least a healthy instance.
• Ensure only specific roles have s3:PutObject permission on the s3 model artifacts route models/model-name/version