Python -> Copy directory while another process removes/adds files from/to it

Question

I work with Windows 10 with Python3.6.

There are two processes:

My process (COPY): Copy a directory X (get as much as possible). (i do it via shutil.copytree)

Foreign process (CREATOR): Deletes and Recreates files in directory X occasionally.

Problem: One of each process 'crashes' occasionally when both processes try to operate on the directory X on the same time. I can handle exceptions in my process but do not want the foreign process to crash.

CREATOR: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: Filepath

I can only change code for my process. Is there a way to copy files (files do not have to be complete) without disturb foreign process?

Background: Multiple foreign processes create test reports containing many html files. I want to copy these to one location during test execution and be able to view the current result from there.

import shutil, os, stat
import multiprocessing as MP
from time import sleep
import random
from traceback import format_exc

work_dir = r"C:\\tmp\\Report"
dest = r"C:\\tmp\\Report_Copy"

def CREATOR():
    while(True):
        if not(os.path.exists(work_dir)):
            os.mkdir(work_dir)
            print("CREATOR: Created %s" % work_dir)

        for root, dirs, files in os.walk(work_dir, topdown=False):
            # remove current files
            for name in files:
                fullName = os.path.join(root, name)
                os.chmod(fullName, stat.S_IWRITE)
                os.remove(fullName)

        print("CREATOR: Removed %s" % work_dir)

        for i in range(1000):
            tmpPath = os.path.join(work_dir, "result%d" % i)
            with open(tmpPath, "w") as f:
                f.write("voodoo"*50000)
        print("CREATOR: Created 100 files")

        sleep(0)

def COPY():
    while(True):
        try:
            sleep(1)


            # SOLUTION FOR WINDOWS
            #######
            os.system("robocopy %s %s /E >nul 2>&1" %(work_dir, dest))
            #######


            #shutil.copytree(work_dir, dest)
            print("COPY: copied %s to %s" % (work_dir, dest))
            shutil.rmtree(dest)
            print("COPY: removed %s" % dest)
        except:
            print("COPY:\n" + format_exc())
            shutil.rmtree(dest)

if __name__ == "__main__":
    if os.path.exists(work_dir):
        shutil.rmtree(work_dir)

    if os.path.exists(dest):
        shutil.rmtree(dest)

    P1 = MP.Process(target=CREATOR)
    P1.start()

    P2 = MP.Process(target=COPY)
    P2.start()

    P1.join()
    P2.join()

COPY process can fail occasionally but CREATOR process should never crash because COPY process is accessing files. The code is only an example. I can not change the code from CREATOR (foreign) process!

Answer 1

This was solved by me by using a OS specific copy tool under Windows.

I used robocopy in COPY Process instead of shutil.copytree ( shutil worked under Linux though).

os.system("robocopy %s %s /E >nul 2>&1" %(src, dest))