Why does Python ignore source files in favor of pyc files sometimes?

Question

I've got a situation where I rapidly change Python source files on disk and, for each change, run them with Python in a subprocess (it's for a mutation testing tool). I have found that in some cases, the Python subprocess either doesn't see the change or ignores it in favor the contents of __pycache__ . Here's an example:

from pathlib import Path
import subprocess

workspace = Path('workspace')
workspace.mkdir(exist_ok=True)

for i in range(3):
    with open(workspace / 'code.py', mode='wt') as f:
        f.write(f"print({i})")
    proc = subprocess.run(
        'python -m code'.split(),
        stdout=subprocess.PIPE,
        cwd='workspace')
    print(i, proc.stdout.decode('utf-8').strip()) 

I would expect this to print:

0 0
1 1
2 2

But generally it prints something like:

0 0 
1 0
2 0

or

0 0
1 1
2 1

That is, at some point the changes I'm writing to disk aren't seen by the Python subprocess.

I've tried all sorts of attempts to flush the changes to disk with os.fsync , flush() on the file handle, and so forth. The only thing that seems to make a difference is putting a substantial time.sleep() after the closing of the file handle.

I could tell Python to not use pycs by setting PYTHONDONTWRITEBYTECODE , but at this point I'd like to know what's going on.

So, am I right that this is just some failure to flush to disk? Or is there perhaps a bug/feature of Python that I'm fighting?

Answer 1

No special flushing is required, but aside from disabling or removing the .pyc , sleep is the only answer: the .pyc doesn't appear older than the .py if the time elapsed between writing them is small enough. “Small enough” depends on the filesystem and OS, but it could easily be a whole second.