我的多处理线程池比单线程实现需要更长的时间来完成任务

Question

I have written an algorithim and am trying to compare performance of diffrent versions. My benchmark function uses a threadpool, but it takes the same time or longer to benchmark than a single-core implementation. I have used pypy and python, versions 3.11 and the result is the same.

Method to benchmark:

def main(print_results=True):
    results = Queue()
    start_time = time.time()

    words = get_set_from_dict_file("usa.txt")
    results.put(f"Total words read: {len(words)}")
    results.put(f"Total time taken to read the file: {round((time.time() - start_time) * 1000)} ms")
    start_time_2 = time.time()

    pairs = getPairs(words)
    results.put(f"Number of words that can be built with 3 letter word + letter + 3 letter word: {len(pairs)}")

    results.put(f"Total time taken to find the pairs: {round((time.time() - start_time_2) * 1000)} ms")

    results.put(f"Time taken: {round((time.time() - start_time) * 1000)}ms")

    if print_results:
        [print(x) for x in results.queue]
    return (time.time() - start_time) * 1000

MultiThreaded Threadpool:

def benchmark(n=1000):
    # start number of threads equal to 90% of cores running main() using multiprocessing, continue until n runs complete
    core_count = os.cpu_count()
    thread_num = floor(core_count * 0.9)
    pool = ThreadPool(thread_num)

    results = pool.map_async(main, [False] * n)
    results = results.get()
    pool.close()
    avg_time_ms = round(sum(results) / len(results))
    # Save best run time and its code as a pickle file in format (time, code)
    # Currently hidden code
    return avg_time_ms, -1

Test:

if __name__ == "__main__":
    print("Do you want to benchmark? (y/n)")
    if input().upper() == "Y":
        print("Benchmark n times: (int)")
        n = input()
        n = int(n) if (n.isdigit() and 0 < int(n) <= 1000) else 100
        start = time.time()
        bench = benchmark(n)
        end = time.time()
        print("\n----------Multi-Thread Benchmark----------")
        print(f"Average time taken: {bench[0]} ms")
        print(f"Best time taken yet: {bench[1]} ms")
        print(f"Total bench time: {end - start:0.5} s")

        start = time.time()
        non_t_results = [main(False) for _ in range(n)]
        end = time.time()
        print("\n----------Single-Thread Benchmark----------")
        print(f"Average time taken: {round(sum(non_t_results) / len(non_t_results))} ms")
        print(f"Total bench time: {end - start:0.5} s")

    else:
        main()

Every time I run it, no matter the number of runs or threads in the pool, the pool never completes faster. Here is an example output:

Do you want to benchmark? (y/n)
y
Benchmark n times: (int)
50

----------Multi-Thread Benchmark----------
Average time taken: 276 ms
Best time taken yet: -1 ms
Total bench time: 2.2814 s

----------Single-Thread Benchmark----------
Average time taken: 36 ms
Total bench time: 1.91 s

Process finished with exit code 0

I expect the threadpool to finish faster.

Answer 1

It turns out I was using threads instead of processes. Thanks to the commentators I was able to understand that ThreadPool is for concurrent processing, and Pool is for parallel processing.

Here was the changed benchmark:

def benchmark(n=1000):
    # start number of threads equal to 90% of cores running main() using multiprocessing, continue until n runs complete
    core_count = os.cpu_count()
    process_num = floor(core_count * 0.9)

    with Pool(process_num) as pool:
        results = pool.map_async(main, [False] * n)
        results = results.get()
    avg_time_ms = round(sum(results) / len(results))
    # Save best run time and its code as a pickle file in format (time, code)
    """..."""
    return avg_time_ms, -1