When to run external commands in Python

Question

In Python, it's possible to run a bash command to achieve something that can be achieved by running equivalent Python code. As a simple example, to cd into a directory and do stuff, you could do either:

import subprocess
subprocess.run('cd directory and do other stuff')

or

import os
os.chdir(path) # path is the path to the directory
#do other stuff

So how do you choose which one to use (I mean in general, not with this simplified example)?

Answer 1

The first version just changes the directory of the subprocess. It has no effect on the original Python process, so it's practically useless. It would only be worthwhile if it were part of multiple commands:

subprocess.run('cd directory; cat filename', shell=True)

The second version changes the directory of the Python process itself.

More generally, if there'a Python function that does the same thing as an external command, you should prefer the function. It's more efficient since it doesn't have to start a new process, it doesn't have to parse the command if you use shell=True , and you get more flexible error handling.

Answer 2

I wouldn't choose the former. It creates a subprocess and changes to the directory inside that subprocess, which then terminates leaving the Python process's current directory the same as it was before the subprocess.run call.

os.chdir actually changes the working directory of the current process. Which seems fine until you start using threads and realise that one thread's current directory can be affected by an os.chdir call from another thread.

For single-threaded processes (the kind an Ingorant Wandered might typically write initially) this presents no problems.

More broadly, expect to use the subprocess module if you need more parallelism than Python can provide alone (research "Python GIL" to find out why naive Python processes can only use one core of a multicore computer).

Answer 3

Niether. subprocess.run('cd directory') just changed the directory of the subprocess which then exited, discarding that changed-directory context. os.chdir(path) is not always insane, just 99% insane. You are better off using os.path or pathlib to track directories and not changing what "current" means throughout the program.

There is an advantage to running external commands, especially if they implement functionality you don't have at the moment. They can also help parallelize your code such as running an external grep and using its results. That has the disadvantage of adding external platform dependencies.

Frequently its only because the programmer happened to understand the shell better than python. Or found a shell example on the net. That's fine for a hobbyist not not a professional code base.

Disclaimer: IMHO