Python os.listdir() doesn't return some files

Question

I have a python function that checks the modified time of a folder using os.stat() and then it does a os.listdir() of that folder.

I have another process which create files in the folder being checked. Sometimes it is observed that three files are created with same timestamp and the folder's stat also has the same timestamp.

When the python function fetches the the files in the same millisecond, it is observed that os.listdir() is providing only 2 of the 3 created files. Why is this so?

The environment is:

OS: Red Hat Enterprise Linux Server release 7.6

Python Version: Python 3.6.8

Filesystem: xfs

Sample code

import os
import sys
import time

filelist=list()
last_mtime=None


def walkover():
    path_to_check="/path/to/check"
    curr_mtime=os.stat(path_to_check).st_mtime_ns
    global last_mtime
    global filelist
    if last_mtime == None or curr_mtime > last_mtime:
        for file in os.listdir(path_to_check):
            if file not in filelist:
                filelist.append(file)
        last_mtime = curr_mtime
        print ("{} modified at {}".format(path_to_check, last_mtime))

The function is invoked to maintain a list of files at a point of time. The if case is present to avoid multiple os.listdir() invokations.

Edit:

The files are ".rsp" files which gets created by ninja when a ".o" is about to get built.

Since my machine has multiple cores (16), the ninja is triggered from cmake with "--parallel 16". This will cause 16 compilations to happen parallely.

Answer 1

I don't know this ninja thing but it seems to me that when writing the first ".rst" file and running the walkover function are executed at approximately the same time the walkover function may be missing out on the creation of the other files. Wether it's buffers or queues or whatever, specifically when parallel processes are in play the order of events can be hard to grasp.