What docker image does GCP AI notebooks actually use? Cannot mount Cloud Filestore from the deeplearning image

Question

I want to launch an AI Platform Notebook instance with a custom container so I can add some dependencies (eg, Python 3.7). However, when I run a custom container, I am unable to mount my data from Cloud Filestore.

Per the docs , I created a custom container using this deep learning image:

gcr.io/deeplearning-platform-release/tf-cpu

I didn't make my own container yet and have added zero customizations; I just plugged gcr.io/deeplearning-platform-release/tf-cpu into the console for my custom instance.

When I try to mount my Cloud Filestore, I get the following errors:

root@c7a60444b0fc:/# mount <IP_ADDRESS>:/streams cfs
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
root@c7a60444b0fc:/# mount -o nolock <IP_ADDRESS>:/streams cfs
mount.nfs: Operation not permitted

Now, when I launch a TensorFlow 1.15 notebook from the console (no customizations), the mount works fine and the environment is different from what I get with the deeplearning image. In particular, the deeplearning image launches as the root user whereas the TF 1.15 instance launches as the jupyter user.

So what image is the GCP AI Notebook actually using? What additional customization does the deeplearning image need to be able to mount a Cloud Filestore?

Answer 1

AI Platform Notebooks environments are provided by container images that you select when creating the instance. In this page you will see the available container image types .

The problem that you are facing is a known issue and it happens when rpc-statd.service is not initialized. I have solved it following these steps:

• Created a Filestore instance
• Created an AI Platform Notebooks instance with Python environment in the same VPC than the Filestore instance

• Entered root mode, installed nfs-common and enabled rpc-statd.service:

   sudo bash sudo apt install nfs-common systemctl enable rpcstatd.service

Using the above commands and following these steps I could properly mount and access Filestore NFS.

In case that you want to further customize a custom container, you should take into account that in order to get access to Cloud Filestore you will have to use a VPN to connect to the same network as your Filestore instance as suggested in this SO post , otherwise you will find a 'Connection timed out' error when calling Filestore's instance internal IP. You could use Cloud VPN to do so.

---