Permission error cloning git repository in temporary folder

Question

Code in question can explain more than I can say in text I think. I've simplified it massively to keep it readable but in essence this is what I'm running.

def main():
    with tempfile.TemporaryDirectory() as td:
        for repo in repos:
            subprocess.run("git clone --mirror {} {}".format(os.path.join(td, repo.path), repo.url)

The cloning part works just fine and goes through the entire list sucessfully. What doesn't is that when the "with... as..." exits it throws the following error(traceback starts at __exit__ of the context manager):

    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\tmp4ew2qffb\\sources\\REPONAME\\objects\\pack\\pack-abd0ff87083dbbcb90f707d8f2d53c730583bb6e.idx'

Running the script as admin doesn't help either. What is going wrong here?

EDIT: I've dug into it and it turns out python 3.7 the TemporaryDirectory cleanup does not support cleaning up read-only files on Windows.

Answer 1

Normally adding at the end of your with tempfile... usage, so before leaving the context something like

for fname in pathlib.Path(td).glob('**/*.*'):  # make all writable for deletion
    fname.chmod(stat.S_IWRITE)

should help.

Note, occasionally I still see PermissionError ("still in use"), not sure yet this is something special in my environment.

Answer 2

This is related but not exactly the same as Deleting read-only directory in Python – one possible work-around from this answer is to explicitly fix the permissions before exiting the TemporaryDirectory() context manager. See akaihola/darker#453 for details. Here's a copy of the implementation:

import os
import sys
from pathlib import Path
from typing import Union

WINDOWS = sys.platform.startswith("win")

def fix_py37_win_tempdir_permissions(dirpath: Union[str, Path]) -> None:
    """Work around a `tempfile` clean-up issue on Windows with Python 3.7

    Call this before exiting a ``with TemporaryDirectory():`` block or in teardown for
    a Pytest fixture which creates a temporary directory.

    See discussion in https://github.com/akaihola/darker/pull/393
    Solution borrowed from https://github.com/python/cpython/pull/10320

    :param dirpath: The root path of the temporary directory

    """
    if not WINDOWS or sys.version_info >= (3, 8):
        return
    for root, dirs, files in os.walk(dirpath):
        for name in dirs + files:
            path = os.path.join(root, name)
            try:
                os.chflags(path, 0)  # type: ignore[attr-defined]
            except AttributeError:
                pass
            os.chmod(path, 0o700)

and here's how to use it in Pytest unit tests or when creating a temporary directory using tempfile :

import pytest

from my_utils import fix_py37_win_tempdir_permissions

@pytest.fixture
def myfixture(tmp_path):
    # setup code here
    yield tmp_path
    fix_py37_win_tempdir_permissions(tmp_path)


def myfunc():
    with TemporaryDirectory() as tmpdir:
        # work on the temporary directory here
        fix_py37_win_tempdir_permissions(tmp_path)