The git process never exits when running as a Python subprocess

Question

Background

I'm writing a Python program to process LFS error messages as I have some repos with missing LFS files in Bitbucket Server. The output when running git lfs fetch --all isn't very helpful when you want to find out which LFS files are missing from a repo. That's why I'm making a tool for it. Maybe it can be contributed back to the git project in some form when I'm done with it?

Code snippet

import subprocess

def git_lfs_fetch(repo_dir) -> list:
    timeout_sec = 120
    try:
        completed_process = subprocess.run(
            ["git", "lfs", "fetch", "--all"], check=False, cwd=repo_dir, 
            capture_output=True, text=True, timeout=timeout_sec, shell=False)
        return completed_process.stderr.split('\n')
    except subprocess.TimeoutExpired as ex:
        print(f'ERROR Could not complete "{ex.cmd}" before timeout of {timeout_sec} s!')
        print(ex.stderr)
        return []

Environment

• Windows 10, 64 bit
• Python 3.10.7
• git version 2.35.2.windows.1
• git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)

Problem

Sometimes, and not in any consistent way, the subprocess.run() method never returns because the git-lfs process never exits. Usually running the git lfs fetch --all command in my test repos takes a few seconds to complete. As a workaround, I added a 2 min timeout to the subprocess.run() call. I figured I could get the output I'm interested in from stderr from the exception, since the git-lfs was done with all it should do. However, that did not help. Python does not seem to be able to kill the git subprocess. I understand from the doc that it sends a SIGKILL to the process and then waits for it to exit. But it never exits, even with the timeout set.

If I manually kill the git-lfs process from the outside I get the expected output printed from ex.stderr so git-lfs sure looks like it is done, and my workaround does what it should.

Fixing my workaround

As I was writing this, in a typical rubberducking fashion, I had an idea.

Since Python fails to forcibly terminate the git subprocess I tried using git-lfs directly, instead of letting git call it. This made the timeout work.

subprocess.run(["git-lfs", "fetch", "--all"], ...)

What I'm looking for

The solution I'm looking for is a way to figure out why git-lfs and consequently git won't terminate properly. Even better would be a fix for that problem.

I have seen similar problems when calling git from Java and C# on both Linux and Windows (several years ago), ie the git command actually completes all it should do but the git process never terminates. So I'm thinking this "hanging" could be a problem in git itself. I really would like to be able to find out why the git-lfs process won't exit. I don't know where to start looking.

Answer 1

By using this method

def git_lfs_fetch(repo_dir) -> list:
    timeout_sec = 120
    try:
        completed_process = subprocess.run(
            ["git", "lfs", "fetch", "--all"], check=False, cwd=repo_dir, 
            capture_output=True, text=True, timeout=timeout_sec, shell=False)
        return completed_process.stderr.split('\n')
    except subprocess.TimeoutExpired as ex:
        print(f'ERROR Could not complete "{ex.cmd}" before timeout of {timeout_sec} s!')
        print(ex.stderr)
        return []