Python: How can I alter the number of threads based on the number of incoming files?

Question

I have a folder ("/home/samples/") and files are continuously written to it by another application. My script is:

def multiThreadedSend():
    for root, subdirs, files in os.walk("/home/samples/"):
        fileList = sorted(files)
    queue = Queue.Queue()
    for x in range(4):
        worker = Thread(target=send_bytes, args=(queue))
        worker.daemon = True
        worker.start()
    for f in fileList:
        queue.put(f)
    queue.join()

I want the number of threads to depend on the number of files. For example, If there are too many files in the folder (say 5000), I want the threads to increase to maybe 40 threads. If there are less than 100 files, I want the number of threads to be 2 or less.

How can this be implemented?

Thank you!

Answer 1

A simple solution is to count the number of files in the directory with something like

 numOfFiles = len([f for f in  os.listdir("/home/samples")])

and then decide what to do before the for loop, something like

threadNum = 40 if numOfFiles > 5000 else 2
for x in range(threadNum):
    [your code]