When using importlib to load a module, can I put it in a package without an __init__.py file?

Question

My Python application loads plugins from a user-specified path (which is not part of sys.path ), according to the importlib documentation :

def load_module(module_name, file_path):
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    return module

plugin_module = load_module("some_plugin", "/path/to/plugins/some_plugin.py")

To make it possible for the user to factor out common functionality between multiple plugins, I want to allow relative imports in the plugins:

from . import plugin_common

def plugin_function(x):
    return plugin_common.something(x)

When implemented like this, I get an ImportError in the plugin:

ImportError: attempted relative import with no known parent package

To my understanding, this is because the some_plugin module is not considered part of a package , and relative imports can therefore not be used (inside some_plugin.py , __name__ is 'user_config' and __package__ is empty).

I can solve this by first loading the surrounding package and then putting the imported module into that package:

load_module("plugins_package", "/path/to/plugins/__init__.py")
plugin_module = load_module("plugins_package.some_plugin", "/path/to/plugins/some_plugin.py")

Now, __name__ is 'plugins_package.some_plugin' , __package__ is 'plugins_package' , and I can use relative imports.

However, this requires the user to put an (empty) __init__.py file in the plugins directory, which I would like to avoid. Since normal packages don't require an __init__.py file (they will be treated as a namespace packages ), it seems like this should be possible.

It seems like it should be possible to create a namespace package dynamically (using importlib ) for plugins_package and using that as package for the imported plugin_module . But I haven't found a way to do this.

So:

• Can I create a namespace package (where I can put plugin_module in) dynamically?
• Can I dynamically create a normal package without the need for an __init__.py file?
• Am I on the wrong track and there is a better way to achieve what I want (relative imports in a module loaded dynamically from outside sys.path )?

Answer 1

I've created an import library that allows you to create a namespace package dynamically: ultraimport .

Check out quickstart example #10 :

 import ultraimport
 plugin_package = ultraimport.create_ns_package('plugins_package.some_plugin', '/path/to/plugins')

With ultraimport you can also load the plugin file and create the namespace package implicitly on the fly like this:

import ultraimport
plugin_module = ultraimport("/path/to/plugins/some_plugin.py", package=1)

This is explained in more detail quickstart example #7 .