How to alter shared memory for SageMaker Docker containers?

Question

I have a Docker image in Elastic Container Registry (ECR). It was created via a simple Dockerfile which I have control over.

The image itself is fine, but I have a problem where the shared memory is insufficient when working inside a container in SageMaker Studio. Therefore I need to raise the shared memory of these containers.

To raise the shared memory of a container, I believe the usual method is to pass the --shm-size argument to the docker run command when starting the container. However, I do not have control over this command, as SageMaker is doing that bit for me. The docs say that SageMaker is running docker run <image> train when starting a container.

Is it possible to work around this problem? Either via somehow providing additional arguments to the command, or specifying something when creating the Docker image (such as in the Dockerfile, deployment script to ECR).

Answer 1

According to this issue there is no option you can use in sagemaker at the moment. If ECS is an option for you, it does support --shm-size option in the task definition.

Answer 2

As pointed out by @rok (thank you!) it is not possible in this situation to pass arguments to docker run , although it would be if switching to ECS.

It is however possible to pass the --shm-size argument to docker build when building the image to push to ECR. This seems to have fixed the problem, albeit it does require a new Docker image to be built and pushed whenever wanting to change this parameter.