在python中多線程時如何增加變量值

Question

我正在嘗試使用多線程制作網絡爬蟲以使其更快。 我想讓每次執行都增加價值。 但有時該值會自行跳過或重復。

import threading
num = 0

def scan():
    while True:
        global num
        num += 1
        print(num)
        open('logs.txt','a').write(str(f'{num}\n'))

for x in range(500):
    threading.Thread(target=scan).start()

結果：

2
2
5
5
7
8
10
10
12
13
13
13
16
17
19
19
22
23
24
25
26
28
29
29
31
32
33
34

預期結果：

1
2
3
4
5
6
7
8
9
10

Answer 1

因此，由於變量num是共享資源，因此您需要對其進行鎖定。 這是按如下方式完成的：

num_lock = threading.Lock()

每次要更新共享變量時，都需要線程先獲取鎖。 一旦獲得了鎖，只有該線程可以訪問更新 num 的值，而當當前線程獲得鎖時，沒有其他線程能夠這樣做。

確保在執行此操作時使用wait或try-finally塊，以保證即使當前線程未能更新共享變量，也會釋放鎖。

像這樣的東西：

num_lock.acquire()
try:
        num+=1
finally:
   num_lock.release()

使用with ：

 with num_lock:
   num+=1

Answer 2

似乎是一種競爭條件。 您可以使用鎖，以便只有一個線程可以獲得特定編號。 使用鎖寫入輸出文件也是有意義的。

這是一個帶鎖的例子。 當然，您不能保證寫入輸出的順序，但每個項目都應該只寫入一次。 在此示例中，我添加了 10000 的限制，以便您可以更輕松地檢查所有內容是否最終都寫入測試代碼中，否則無論何時中斷它，都很難驗證某個數字是被跳過還是只是在等待用於寫輸出的鎖。

該my_num不共享，那么你已經聲明它里面后with num_lock部分，你可以自由地釋放鎖（保護共享num ），然后繼續使用my_num在外面with其他線程可以訪問鎖定以索取自己的價值。 這最大限度地減少了持有鎖的持續時間。

import threading

num = 0
num_lock = threading.Lock()
file_lock = threading.Lock()    

def scan():
    global num_lock, file_lock, num
    
    while num < 10000:
        with num_lock:
            num += 1
            my_num = num

        # do whatever you want here using my_num
        # but do not touch num

        with file_lock:
            open('logs.txt','a').write(str(f'{my_num}\n'))
        
threads = [threading.Thread(target=scan) for _ in range(500)]

for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

Answer 3

除了threading.Lock之外還有一個重要的標注：

• 使用join使父線程等待分叉線程完成。
• 沒有這個，線程仍然會競爭。

假設我在線程完成后使用num ：

import threading

lock, num = threading.Lock(), 0


def operation():
    global num
    print("Operation has started")
    with lock:
        num += 1


threads = [threading.Thread(target=operation) for x in range(10)]
for t in threads:
    t.start()

for t in threads:
    t.join()

print(num)

沒有連接，不一致（9 打印一次，否則 10）：

Operation has started
Operation has started
Operation has started
Operation has started
Operation has startedOperation has started

Operation has started
Operation has started
Operation has started
Operation has started9

使用 join，其一致：

Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
10