Azure Databricks cluster init script - Install wheel from mounted storage

Question

I have a python wheel uploaded to an azure storage account that is mounted in a databricks service. I'm trying to install the wheel using a cluster init script as described in the databricks documentation .

My storage is definitely mounted and my file path looks correct to me. Running the command display(dbutils.fs.ls("/mnt/package-source")) in a notebook yields the result:

path: dbfs:/mnt/package-source/parser-3.0-py3-none-any.whl
name: parser-3.0-py3-none-any.whl

I have tried to install the wheel from a cluster init file using this command:

/databricks/python/bin/pip install "dbfs:/mnt/package-source/parser-3.0-py3-none-any.whl"

but the cluster fails to start. It's logs give me an error saying it can't find the file:

WARNING: Requirement 'dbfs:/mnt/package-source/parser-3.0-py3-none-any.whl' looks like a filename, but the file does not exist
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/dbfs:/mnt/package-source/parser-3.0-py3-none-any.whl'

I have also tried it this way:

/databricks/python/bin/pip install /mnt/package-source/parser-3.0-py3-none-any.whl

but I get a similar error:

WARNING: Requirement '/mnt/package-source/parser-3.0-py3-none-any.whl' looks like a filename, but the file does not exist
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/mnt/package-source/parser-3.0-py3-none-any.whl'

I've even tried using relative paths such as ../../mnt/package-source/... but to no avail. Can anyone tell me what I'm doing wrong please?

Related question: Azure Databricks cluster init script - install python wheel

Answer 1

I got it working using a relative path. It turns out ../../mnt/ wasn't the correct path. It worked using ../../../dbfs/mnt/ . It just took a bit of exploring the file system using the bash ls command to find it.

For anyone else experiencing the same problem, I suggest starting with something like this in a notebook:

%%sh
ls ../../../