Python Threading: Multiline Progress Report

Question

I'm writing this program that downloads multiple files simultaneously using threads and reports the progress for each file on the screen. I'm using one different thread to download each file. That is no problem:

for file_url, filename in zip(file_urls, filenames):
    th = Thread(target=download_file, args=(file_url, filename))
    threads.append(th)
    th.start()

for thread in threads:
    thread.join()

The problem is I don't know anyways that I can print each file progress report on the screen that will look like this:

    "Downloading 'example1.zip':  54366 bytes of 2240799  70.7%" 
    "Downloading 'example2.zip':  31712 bytes of 1924639  64.7%" 
    "Downloading 'example3.zip':  21712 bytes of 3224979  34.7%" 

The following snippet is for a single line progress report:

def chunk_report(bytes_so_far, total_size, filename):

    percent = float(bytes_so_far) / total_size
    percent = round(percent * 100, 2)
    print "Downloading '{0}':  {1} of {2}  {3:3.2g}% \r".format(filename,
                                        bytes_so_far, total_size, percent),

and the output would look like this:

    "Downloading 'example2.zip':  31712 bytes of 1924639  64.7%" 

each time a thread call this function it updates the screen for the file that, the thread is downloading.

So, the question is how do I print multiline progress report like the one that I illustrated above in python?

Thanks in advance.

Answer 1

I would use Queue to report progress to a reporting thread:

1. Create a Queue
2. Spawn each downloading thread passing the Queue as an argument
3. Have each downloading put progress messages to the Queue
4. Spawn a reporting thread which reads progress messages from the Queue and updates the display

A simulated example:

import threading
import time
import random
import Queue
import sys

# a downloading thread
def worker(path, total, q):
  size = 0
  while size < total:
    dt = random.randint(1,3)
    time.sleep(dt)
    ds = random.randint(1,5)
    size = size + ds
    if size > total: size = total
    q.put(("update", path, total, size))
  q.put(("done", path))

# the reporting thread
def reporter(q, nworkers):
  status = {}
  while nworkers > 0:
    msg = q.get()
    if msg[0] == "update":
      path, total, size = msg[1:]
      status[path] = (total, size)
      # update the screen here
      show_progress(status)
    elif msg[0] == "done":
      nworkers = nworkers - 1
  print ""

def show_progress(status):
  line = ""
  for path in status:
    (total, size) = status[path]
    line = line + "%s: %3d/%d   " % (path, size,total)
  sys.stdout.write("\r"+line)
  sys.stdout.flush()

def main():
  q = Queue.Queue()
  w1 = threading.Thread(target = worker, args = ("abc", 30, q) )
  w2 = threading.Thread(target = worker, args = ("foobar", 25, q))
  w3 = threading.Thread(target = worker, args = ("bazquux", 16, q))
  r = threading.Thread(target = reporter, args = (q, 3))
  for t in [w1,w2,w3,r]: t.start()
  for t in [w1,w2,w3,r]: t.join()

main()