Async execution of commands in Python

Question

Requirement - I want to execute a command that uses ls, grep, head etc using pipes (|). I am searching for some pattern and extracting some info which is part of the query my http server supports.

The final output should not be too big so m assuming stdout should be good to use (I read about deadlock issues somewhere) Currently, I use popen from subprocess module but I have my doubts over it.

• how many simultaneous popen calls can be fired.
• does the result immediately come in stdout? (for now it looks the case but how to ensure it if the commands take long time)
• how to ensure that everything is async - keeping close to single thread model?

I am new to Python and links to videos/articles are also appreciated. Any other way than popen is also fine.

Answer 1

You could use os.listdir or os.walk instead of ls , and the re module instead of grep .

Wrap everything up in a function, and use eg the map method from a multiprocessing.Pool object to run several of those in parallel. This is a pattern that works very well.

In Python3 you can also use Executors from concurrent.futures in a similar way.