No space left on device [Amazon SageMaker]

Question

I am working on training my model in P2.xlarge instance. When I download a dataset, I get the following error: "Exception during downloading or extracting: [Errno 28] No space left on device"\ I checked that P2.xlarge has 61GiB storage, translating to 64GB. I hardly have 5GB worth of data in my instance. Could you please let me know how to proceed?

Answer 1

by default, notebook instances have 5GB of storage, no matter what type they are. You can also use 20GB of non-persistent storage in /tmp.

https://docs.aws.amazon.com/sagemaker/latest/dg/howitworks-create-ws.html

Hope this helps.

Answer 2

Thank you for using Amazon SageMaker.

The 61GiB on P2.xlarge is RAM storage which is different from persistent storage you get with any Notebook Instance. At this time, by default each Notebook Instance have 5 GB of storage regardless of instance type. Please note that only data stored in this 5 GB storage is persisted across Notebook Instance restarts, you can also use ~10* GB of non-peristent storage in /tmp but that data will not be persisted across Notebook Instance restarts.

On the side note, you can also use EFS file system with SageMaker to have more space with Notebook Instance. There is blog on how to add an EFS mount with Lifecycle actions, here Mount an EFS file system to an Amazon SageMaker notebook (with lifecycle configurations) .

Let us know if there is any other way we can be of assistance.

• Disclaimer : The available size for non-persistent storage in /tmp may change over time.

• Note : This looks like duplicate of post on AWS forums, here https://forums.aws.amazon.com/thread.jspa?messageID=858201 .

Thanks,
Neelam

Answer 3

Even when you add extra storage, docker on SageMaker Notebook instances uses / mounted on the default block storage. You can configure SageMaker to use another directory (inside your extra storage device) by following instructions here . Usually, your home directory ( /home/ec2-user/SageMaker ) is mounted on this extra storage

1. Create ~/.sagemaker/config.yaml

2. Set a new container root where you have more space

local:
  container_root: /home/ec2-user/SageMaker/temp