当父目录位于 sys.path 中时，Python 无法始终找到模块

Question

I'm currently working on a project where I need to import self made modules that could be anywhere on the filesystem. But python can't seem to consistently import them.

All the modules have the same name which is why I couple them with their parent directory, like so: xyz.abc for a module called abc.py in a folder called xyz . xyz's parent directory is added to path. To import them later on, I use importlib.import_module(module_name) where module name is xyz.abc .

To test my code, I am using python's tempfile lib to create temporary directories and temporary files. ( /tmp on my system is cleaned at reboot and not automatically.)

I've tried adding an __init__.py file, it doesn't change anything. I've also tried os.sync() to force write the file to my disk thinking it might be a race problem but it didn't help.

Here's the code where it breaks:

if parent not in sys.path:
    sys.path.insert(0, parent)
module_name = os.path.basename(dir_path) + "." + filename.split(".")[0]
return importlib.import_module(module_name)

The tree structure of the directory is:

/tmp/tmp9tp_1j_q/
└── tmprvbxhk3t.py

And sys.path has '/tmp' in it when I print it right before module_name =... .

When executing the code, sometimes it works and sometimes it doesn't . I get: ModuleNotFoundError: No module named 'tmpfjig62pf'

I'm thinking maybe sys.path isn't internally updated directly after an insert.

I need some help to solve this problem.

Answer 1

The problem isn't in sys.path , but it's close. It is the Finder object in sys.meta_path that uses sys.path and sys.path_hooks to look for modules, and sys.meta_path has internal caches that aren't "refreshed" automatically in such a situation.

So, to successfully import a module that's been created after the interpreter began execution, the internal caches of sys.meta_path should be invalidated.

That can be achieved by calling importlib.invalidate_caches() after creating the test modules.