Python Dictionary million records process via threads

Question

There is Python dictionary with millions records incremental order that need to be processed by threads.

data = {}
data[number] = name

The numbers are in incremental order, they go from 1 to 1 million

The threards are called tested via

First option backfires as it only processes certain threads defined by max_tries.

for i in xrange(1, max_tries + 1):
    name = data.get(str(i))

    current = MainThread(name)
    check_results.append(current)
    current.start()

Second one executes a million records at a time.

for item in data:
    name = data[item]

    current = MainThread(name)
    check_results.append(current)
    current.start()

How would you solve it.

Answer 1

I would use a ThreadPoolExecutor . I haven't tested the following codes, so please treat them as pseudo codes.

import concurrent.futures

def my_processing_function():
    pass

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:

    future_objects = []
    for item in data:
        name = data[item]
        future_objects.append(executor.submit(my_processing_function, name, 60))

    for future in concurrent.futures.as_completed(future_objects):
        try:
            res = future.result()
        except Exception as exc:
            print('exception: {}'.format(exc))
        else:
            print("Result: {}".format(res))