如何将本地模块导入 azure databricks 笔记本？

Question

I'm trying to use a module in databrick's notebook but I am completely blocked. I'd like to execute the following command or anything similar which allow my to make instances of MyClass

from mypackage.mymodule import MyClass

Following databrick's documentation I have develop a python package with a single module locally as follows:

mypackage
|- __init__.py
|- setup.py
|- mymodule.py

Then run python setup.py bdist_wheel obtaining a .whl file. The directory ends up being

mypackage
|- build
   |- ... whatever
|- src.egg-info
   |- ... whatever
|- dist
   |- src-0.1-py3-none-any.whl
|- __init__.py
|- setup.py
|- mymodule.py

From here I've uploaded the .whl file into the Workspace following the instructions. But now I'm not able to import MyClass into any notebook.

I've tried all approches below:

• upload the .whl with and without a name.
• upload the .whl installing it into the cluster and not.
• use import mypackage
• use dbutils.library.install('dbfs:/path/to/mypackage.whl/') (which returns True ) and then use import ...
• instead of uploading a .whl , create the package folder in the same directory as the notebook.
• Upload to my folder and to Shared folder
• all combinations of the above. f.ex: uploading with different name and use import differentname

This is driving my crazy. I its such a simple task which I can achive easily with regular notebooks.

Answer 1

I've solved this by using python's egg instead of wheel . python setup.py bdist_egg will create an egg which you can install following databricks docs. I don't know why wheel doesn't work...

Answer 2

With the the introduction of support for arbitrary files in Databricks Repos , it is now possible to import custom modules/packages easily, if the module/package resides in the linked git repo.

First,

1. Make sure Repos for Git integration is enabled.
2. Make sure support for arbitrary files is enabled.

Both of these can be enabled from Settings -> Admin Console -> Workspace Settings.

Then, with the following directory structure in the git repo,

.
├── mypackage
│   ├── __init__.py
│   └── mymodule.py
└── test_notebook

it is possible to import the module mymodule in the package mypackage from test_notebook simply by executing the following statement:

# This is test_notebook in the above filetree
from mypackage.mymodule import MyClass