Importing python modules from multiple directories

Question

I have a python 2.6 Django app which has a folder structure like this:

/foo/bar/__init__.py

I have another couple directories on the filesystem full of python modules like this:

/modules/__init__.py
/modules/module1/__init__.py
/other_modules/module2/__init__.py
/other_modules/module2/file.py

Each module __init__ has a class. For example module1Class() and module2Class() respectively. In module2 , file.py contains a class called myFileClass() .

What I would like to do is put some code in /foo/bar/__init__.py so I can import in my Django project like this:

from foo.bar.module1 import module1Class
from foo.bar.module2 import module2Class
from foo.bar.module2.file import myFileClass

The list of directories which have modules is contained in a tuple in a Django config which looks like this:

module_list = ("/modules", "/other_modules",)

I've tried using __import__ and vars() to dynamically generate variables like this:

import os
import sys 

for m in module_list:
    sys.path.insert(0, m)
    for d in os.listdir(m):
        if os.path.isdir(d):
            vars()[d] = getattr(__import__(m.split("/")[-1], fromlist=[d], d)

But that doesn't seem to work. Is there any way to do this?

Thanks!

Answer 1

I can see at least one problem with your code. The line...

if os.path.isdir(d):

...won't work, because os.listdir() returns relative pathnames, so you'll need to convert them to absolute pathnames, otherwise the os.path.isdir() will return False because the path doesn't exist (relative to the current working directory), rather than raising an exception (which would make more sense, IMO).

The following code works for me...

import sys
import os

# Directories to search for packages
root_path_list = ("/modules", "/other_modules",)

# Make a backup of sys.path
old_sys_path = sys.path[:]

# Add all paths to sys.path first, in case one package imports from another
for root_path in root_path_list:
    sys.path.insert(0, root_path)

# Add new packages to current scope
for root_path in root_path_list:
    filenames = os.listdir(root_path)
    for filename in filenames:
        full_path = os.path.join(root_path, filename)
        if os.path.isdir(full_path):
            locals()[filename] = __import__(filename)

# Restore sys.path
sys.path[:] = old_sys_path

# Clean up locals
del sys, os, root_path_list, old_sys_path, root_path, filenames, filename, full_path

---

Update

Thinking about it, it might be safer to check for the presence of __init__.py , rather than using os.path.isdir() in case you have subdirectories which don't contain such a file, otherwise the __import__() will fail.

So you could change the lines...

        full_path = os.path.join(root_path, filename)
        if os.path.isdir(full_path):
            locals()[filename] = __import__(filename)

...to...

        full_path = os.path.join(root_path, filename, '__init__.py')
        if os.path.exists(full_path):
            locals()[filename] = __import__(filename)

...but it might be unnecessary.

Answer 2

We wound up biting the bullet and changing how we do things. Now the list of directories to find modules is passed in the Django config and each one is added to sys.path (similar to a comment Aya mentioned and something I did before but wasn't too happy with). Then for each module inside of it, we check for an __init__.py and if it exists, attempt to treat it as a module to use inside of the app without using the foo.bar piece.

This required some adjustment on how we interact with the modules and how developers code their modules (they now need to use relative imports within their module instead of the full path imports they used before) but I think this will be an easier design for developers to use long-term.

We didn't add these to INSTALLED_APPS because we do some exception handling where if we cannot import a module due to dependency issues or bad code our software will continue running just without that module. If they were in INSTALLED_APPS we wouldn't be able to leverage that flexibility on when/how to deal with those exceptions.

Thanks for all of the help!