How do I load a Python module with custom globals using importlib?

Question

I'm trying to put together a small build system in Python that generates Ninja files for my C++ project. Its behavior should be similar to CMake; that is, a bldfile.py script defines rules and targets and optionally recurses into one or more directories by calling bld.subdir() . Each bldfile.py script has a corresponding bld.File object. When the bldfile.py script is executing, the bld global should be predefined as that file's bld.File instance, but only in that module's scope .

Additionally, I would like to take advantage of Python's bytecode caching somehow, but the .pyc file should be stored in the build output directory instead of in a __pycache__ directory alongside the bldfile.py script.

I know I should use importlib (requiring Python 3.4+ is fine), but I'm not sure how to:

1. Load and execute a module file with custom globals.
2. Re-use the bytecode caching infrastructure.

Any help would be greatly appreciated!

Answer 1

I studied importlib 's source code and since I don't intend to make a reusable Loader , it seems like a lot of unnecessary complexity. So I just settled on creating a module with types.ModuleType , adding bld to the module's __dict__ , compiling and caching the bytecode with compile , and executing the module with exec . At a low level, that's basically all importutil does anyway.

Answer 2

Injecting globals into a module before execution is an interesting idea. However, I think it conflicts with several points of the Zen of Python . In particular, it requires writing code in the module that depends on global values which are not explicitly defined, imported, or otherwise obtained - unless you know the particular procedure required to call the module.

This may be an obvious or slick solution for the specific use case but it is not very intuitive. In general, (Python) code should be explicit. Therefore, I would go for a solution where parameters are explicitly passed to the executing code. Sounds like functions? Right:

bldfile.py

def exec(bld):
    print('Working with bld:', bld)
    # ...

calling the module:

# set bld

# Option 1: static import
import bldfile
bldfile.exec(bld)

# Option 2: dynamic import if bldfile.py is located dynamically
import importlib.util
spec = importlib.util.spec_from_file_location("unique_name", "subdir/subsubdir/bldfile.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
module.exec(bld)

That way no code (apart from the function definition) is executed when importing the module. The exec function needs to be called explicitly and when looking at the code inside exec it is clear where bld comes from.

Answer 3

It is possible to overcome the lack of possibility by using dummy module, which would load its globals .

#service.py 
module = importlib.import_module('userset')
module.user = user 
module = importlib.import_module('config')

#config.py
from userset import *
#now you can use user from service.py