Python import classes from modules dynamically

Question

I'm working with an API which has many sub components. I'm making a toolkit to interact with this API and I would like to make it so that I can dynamically load some of the classes based on variables, for example, loading all classes which have a variable api_module = True set inside.

.
├── APIModules
│   ├── ccu.py
│   ├── __init__.py
│   └── OpenAPI.py

The CCU class within the ccu file

class CCU:
    api_module = True
    actions    = ['post_new_purge','get_purge_queue','get_purge_status']

This will help me to dynamically document and grow the toolkit as more classes are added without having to maintain lists of imports each time a new sub component is added. Is there an easy way to do this?

I know I can get a list of all the python files in the APIModules directory with a glob, but I then want to load and check any classes within the files without knowing what they might be called.

import os
import glob
modules = glob.glob(os.path.dirname(__file__)+"/*.py")

Answer 1

Since you would need to import the class to read its attributes, and you are trying to figure out if you should import it in the first place, you have a catch 22 here.

This leaves you with the approach of parsing the .py file with other methods and deciding based on that whether to import the module in question or not. But this approach is also messy and too complicated.

The most simple approach, I think, would be to change the point of view completely. Instead of having api_module = True inside every module, you can just have one list of active modules inside your main application, and decide based on that:

api_modules = ["module_1", "module_2", "module_3"]

This reduces the amount of maintenance (you only have to maintain one single line of code) and provides the same amount of flexibility.

Answer 2

You could use metaclasses along with a registry of classes to track any new classes that are added to the system.

There's a walkthrough here that should give you the help you need http://effbot.org/zone/metaclass-plugins.htm

Whether or not it's a good idea is another question (I think sqlalchemy uses this pattern for the declarative base and it works well).

# from http://effbot.org/zone/metaclass-plugins.htm
registry = [] # list of subclasses

class Plugin(object):
    class __metaclass__(type):
        def __init__(cls, name, bases, dict):
            type.__init__(name, bases, dict)
            registry.append((name, cls))

# in your plugin modules
class SpamPlugin(Plugin):
    pass

class BaconPlugin(Plugin):
    pass

# in your plugin loader
# import all plugin modules

# loop over registered plugins
for name, cls in registry:
    if cls is not Plugin:
        print name, cls
        # here you could check various attributes of the class to
        # see if it was one you wanted to use

Edit I think I've sort of misunderstood slightly - you don't want to have to import the modules one by one either - but you could write something to import all of the files in the api folder. Then each of the classes would inherit from the base API class that maintains the registry. When you want to work with anything you go to the registry to find the classes you're interested in.