Why does importing fail after creating a module under Python 3 on Windows?

Question

The following code tries to create-then-import two modules:

# coding: utf-8

import os
import time

# Remove the modules we're about to create if they already exist
def force_unlink(name):
    try:
        os.unlink(name)
    except OSError:
        pass
force_unlink("print1.py")
force_unlink("print1.pyc")
force_unlink("print2.py")
force_unlink("print2.pyc")
time.sleep(1)

# Create module 1 and module 2, then try to import them just afterwards
print("Creating module 1...")
with open("print1.py", "wb+") as fd:
    fd.write(b'print("Imported module 1")')
import print1
print("Creating module 2...")
with open("print2.py", "wb+") as fd:
    fd.write(b'print("Imported module 2")')
import print2

On Windows, both imports work under Python 2 (2.7), but not under Python 3 (3.5 and 3.6):

$ python2 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Imported module 2
$ python3 reproduce.py
Creating module 1...
Imported module 1
Creating module 2...
Traceback (most recent call last):
  File "reproduce.py", line 26, in <module>
    import print2
ImportError: No module named 'print2'

Adding time.sleep(5) before each import printX call makes it work.

Why is that?

Note: This is a simpler version of an issue I'm trying to figure out.

Answer 1

I think I know what is going on. The new Python 3 import machinery caches the filenames it finds in directories. It will reload the cache when the mtime , the modification time, of the directory changes.

See the importlib._bootstrap_external.FileFinder.find_spec() method implementation , which contains:

try:
    mtime = _path_stat(self.path or _os.getcwd()).st_mtime
except OSError:
    mtime = -1
if mtime != self._path_mtime:
    self._fill_cache()
    self._path_mtime = mtime

Here _path_stat is just a os.stat() call, but localised to avoid imports. The _fill_cache() method executes the os.listdir() call.

On some Windows filesystems, the resolution of mtime is notoriously low, up to 2 seconds. For your case, the resolution is apparently still low enough for the cache to not be updated by the time you try to load the second module. Although the NTFS filesystem can record times in 100ns increments, in practice the limiting factor there appears to be the Windows system clock, which I understand is usually limited to a resolution of 15ms. So if you write print2.py within 15ms of writing print1.py , then Python won't notice.

Python does give you the means to clear this cache; use the importlib.invalidate_caches() method ; this will reset the _path_mtime attribute on the FileFinder instance back to -1 , forcing a new _fill_cache() call.

As the function's documentation states:

This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module's existence.