Threading subprocesses in Python

Question

If I use Python's Threading library can I accomplish a batch of subprocess processes faster? Let's say for instance I need to convert 100 .wav files to .mp3 files. If I wrap 'ffmpeg' in a Python script that used threading could I accomplish the task much faster? Does threading enable me to actually use all 8 threads available in my i7?

I recently stumbled across a Python script that pings a list of hosts but utilizes threading to make things faster. The script seemed succinct and was easy enough for a beginner such as myself to read and understand. So this is where my other question arises: What is all the talk about running parallel tasks being so complex? Maybe I don't understand parallel and parallel != threading. If it is that simple then why wouldn't people use threading for any and all batch conversions when running on a modern processor?

Answer 1

Here is a simple and working solution for your case. I use it often and it proves to be very useful! This code creates as many threads as cores and lets them execute a (large) number of tasks (in this case, calling a shell program):

import Queue
import threading
import multiprocessing
import subprocess

q = Queue.Queue()
for i in range(30): #put 30 tasks in the queue
    q.put(i)

def worker():
    while True:
        item = q.get()
        #execute a task: call a shell program and wait until it completes
        subprocess.call("echo "+str(item), shell=True) 
        q.task_done()

cpus=multiprocessing.cpu_count() #detect number of cores
print("Creating %d threads" % cpus)
for i in range(cpus):
     t = threading.Thread(target=worker)
     t.daemon = True
     t.start()

q.join() #block until all tasks are done

Answer 2

如果使用正确的程序，使你的程序多线程化线程肯定会使它更快，但是你无法真正控制你的处理器的数量，或你的应用程序将使用多少核心，因为它全部由操作系统处理 ，python中的多处理模块是你可以采取另一种方法，但如果你在尝试这些模块之前更详细地阅读这个主题会更好，我推荐这个教程 。

Answer 3

你应该在python中使用多处理模块，因为你可以在你的计算机中使用多个核心。 在python中尝试这个多进程模块