Shutil.move inside a recursive function

Question

I wrote this function to achieve a similar behavior as in Windows when moving files and directories. In particular, objects should be overwritten.

from pathlib import Path
import shutil
import os


def move_anyhow(source: Path | str, dest: Path | str) -> Path:
    """
    Move source (directory or file) and overwrite files with same name in dest if exists.
    """
    try:
        shutil.move(source, dest)
    except shutil.Error:
        if source.is_file():
            shutil.move(source, dest / source.name)
        else:
            for path in source.iterdir():
                move_anyhow(path, dest / source.name)
            os.rmdir(source)
    return dest / source.name

I took a recursive approach to moving nested source directories like this one

.../source/

dir_A/
  dir_B/
    file_X

to destination

.../dest/

dir_A/
  dir_B/
    file_X
    file_Y

On production I get a PermissionError now and then which looks like this:

PermissionError: [Errno 13] Permission denied: '/delivery/post/01_FROM_CF/W22_FW/50479944_003' -> '/delivery/post/01_FROM_CF/ERROR/W22_FW/50479944_003'
  File "shutil.py", line 813, in move
    os.rename(src, real_dst)

OSError: [Errno 39] Directory not empty: '/delivery/post/01_FROM_CF/W22_FW/50479944_003'
  File "ors/path.py", line 34, in move_anyhow
    shutil.move(source, dest)
  File "shutil.py", line 831, in move
    rmtree(src)
  File "shutil.py", line 728, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "shutil.py", line 726, in rmtree
    os.rmdir(path)

All files were moved but the empty source folder remained. I can't reproduce this error locally. So I my first guess was that this is a server issue. Still, I wonder if the nested approach could cause this error.

So I guess my question is whether an catched shutil.move error can block another shutil.move operation of a file inside the source directory.

Answer 1

The problem will probably be in the timing of files deletion and directory deletion. I would try to cancel the immediate deletion of directories and I would delete the directories only at the end.

Remove the command line:

os.rmdir(source)

and when the file transfer is complete, then call the command

shutil.rmtree(source)

Good luck.