python importlib.import_module requires explicitly imported module

Question

i want to implement sort of plugin architecture that dynamically loads modules and calls a function from them

for instance plugin code looks like (in file "foo_func.py")

foo_local = []

def foo_add(a: int, b: int) -> int:
    c = a + b
    foo_local.append(c)
    return c


def foo_print():
    print(foo_local)

i need to support two plugins with the same code but with different memory state, so i created directory structure like this:

<ROOT_PROJECT>
app.py
bar/
  apple/
    foo/
      foo_func.py
      __init__.py
  orange/
    foo/
      foo_func.py
      __init__.py

code in "apple" and "orange" folders is the same.

then in app file i try to load modules and invoke functions from them

import importlib

from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print

apple = importlib.import_module('bar.apple.foo')
orange = importlib.import_module('bar.orange.foo')

apple_foo = getattr(apple, 'foo_func')
orange_foo = getattr(orange, 'foo_func')

apple_foo_add_my = getattr(apple_foo, 'foo_add')
apple_foo_print_my = getattr(apple_foo, 'foo_print')

apple_foo_add_my(1, 2)
apple_foo_print_my()

and this works fine, but you see these import lines at the top

from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print

they are not used in code (even pycharm complains about it)

but if i try to comment code and run it - then failure

AttributeError: module 'bar.apple.foo' has no attribute 'foo_func'

why?

I suppose normal plugins should deal only with "importlib.import_module" and "getattr" and it must be enough?

what is wrong here?

Answer 1

Let's switch completely to direct imports for this explanation, because:

import something.whatever as name

is the same as:

name = importlib.import_module("something.whatever")

So let's rewrite your apple code:

apple = importlib.import_module('bar.apple.foo')
apple_foo = getattr(apple, 'foo_func')

will become:

import bar.apple.foo as apple
apple_foo = apple.foo_func

Now, the first line loads bar.apple.foo as a module . In case of packages, this means importing package's __init__.py code. And treating it as a module itself.

And what's the code in the package's init? Usually nothing. That's why the name lookup fails.

However, when you do any import my_package.whatever, the package gets its insides checked and the name becomes visible. You're basically pre-loading the module for interpreter to look at.

Why is pycharm giving you not used suggestion? Because it's not used as a variable anywhere. You're only using a side-effect + pycharm doesn't analyze strings for imports or attributes.

---

Visual example, with a part of standard library:

>>> import xml
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> xml.etree
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'xml' has no attribute 'etree'
>>> 
>>> import xml.etree
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'etree']

And another example, what happens if there are multiple modules in the package:

>>> import dateutil
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version']

but:

>>> import dateutil.parser
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_common', '_version', 'parser', 'relativedelta', 'tz']

All sub-modules are now visible and usable with their qualified name.

---

tl;dr: import my_package == only my_package/__init__.py is looked at. import my_package.whatever == python now knows it's a package and registers its insides, all modules of my_package are visible and usable.