How to create a serverless endpoint in aws sagemaker?

Question

I am recreating an endpoint currently working in SageMaker for inference by a serverless endpoint. I am using one of the AWS maintained baseimages: 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.5-cpu-py3 .

Everything works when I choose non-serverless (ie provisioned option for endpoint configuration), but when I try to create one with the serverless option, it fails. The error messages from the logs in cloud watch are below, starting with python and log4j error at the end.

OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k. python: can't open file '/usr/local/bin/deep_learning_container.py': [Errno 13] Permission denied. Warning: MMS is using non-default JVM parameters: -XX:-UseContainerSupport Failed to reap children process, log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: logs/mms_log.log (No such file or directory)

I have set memory to the maximum allowed memory size of 6GB for the serverless option. Why am I getting this error?

Answer 1

How big is your model? If you are using the SageMaker SDK, have you set model_server_workers in your PyTorchModel?

pytorch_model = PyTorchModel(model_data='', 
                             role=role, 
                             .
                             .
                             .
                             model_server_workers=1
                            )

Kindly test using just 1 worker or leave model_server_workers as None, as that set one worker per vCPU.