使用 Queue() 进行多处理：TypeError: can't pickle _thread.lock objects

Question

This is my code:

def search(page, sort, start, end, q):
    print(mp.current_process())
    results = req.request_buff_market_page(page, sort, start, end)

    for i in results:
        if(i[1] >= 20 and 'Souvenir' not in i[0]):
            q.put(i)


if __name__ == '__main__':
    page = 1
    q = queue.Queue()
    with mp.Pool(processes=mp.cpu_count()) as pool:
        print(page)
        pool.starmap(search, (page, 'asc', 200, 205, q))
        page += 1

    while not(q.empty()):
        print(q.get())

Output:

 Traceback (most recent call last):
      File "C:\Users\Andrew\Desktop\ultralight\mp_buff.py", line 33, in <module>
        pool.starmap(search, (page, 'asc', 200, 205, q))
    .........
    TypeError: can't pickle _thread.lock objects

I'm not sure what is causing this error. The error occurs on this line:

pool.starmap(search, (page, 'asc', 200, 205, q))

EDIT: Changed my code to use multiprocessing.Process()

 for i in range(mp.cpu_count()):
     workers.append(mp.Process(target=search, args=(page, 'asc', 200, 230, q_search)))
     page += 1
 for i in workers: i.start()
 for i in workers: i.join()

Answer 1

You are treating your multiprocessing pool as a multithreading pool, which is not the same thing. Look into multiprocessing.pool.ThreadPool .