在python中多线程时如何增加变量值

Question

我正在尝试使用多线程制作网络爬虫以使其更快。 我想让每次执行都增加价值。 但有时该值会自行跳过或重复。

import threading
num = 0

def scan():
    while True:
        global num
        num += 1
        print(num)
        open('logs.txt','a').write(str(f'{num}\n'))

for x in range(500):
    threading.Thread(target=scan).start()

结果：

2
2
5
5
7
8
10
10
12
13
13
13
16
17
19
19
22
23
24
25
26
28
29
29
31
32
33
34

预期结果：

1
2
3
4
5
6
7
8
9
10

Answer 1

因此，由于变量num是共享资源，因此您需要对其进行锁定。 这是按如下方式完成的：

num_lock = threading.Lock()

每次要更新共享变量时，都需要线程先获取锁。 一旦获得了锁，只有该线程可以访问更新 num 的值，而当当前线程获得锁时，没有其他线程能够这样做。

确保在执行此操作时使用wait或try-finally块，以保证即使当前线程未能更新共享变量，也会释放锁。

像这样的东西：

num_lock.acquire()
try:
        num+=1
finally:
   num_lock.release()

使用with ：

 with num_lock:
   num+=1

Answer 2

似乎是一种竞争条件。 您可以使用锁，以便只有一个线程可以获得特定编号。 使用锁写入输出文件也是有意义的。

这是一个带锁的例子。 当然，您不能保证写入输出的顺序，但每个项目都应该只写入一次。 在此示例中，我添加了 10000 的限制，以便您可以更轻松地检查所有内容是否最终都写入测试代码中，否则无论何时中断它，都很难验证某个数字是被跳过还是只是在等待用于写输出的锁。

该my_num不共享，那么你已经声明它里面后with num_lock部分，你可以自由地释放锁（保护共享num ），然后继续使用my_num在外面with其他线程可以访问锁定以索取自己的价值。 这最大限度地减少了持有锁的持续时间。

import threading

num = 0
num_lock = threading.Lock()
file_lock = threading.Lock()    

def scan():
    global num_lock, file_lock, num
    
    while num < 10000:
        with num_lock:
            num += 1
            my_num = num

        # do whatever you want here using my_num
        # but do not touch num

        with file_lock:
            open('logs.txt','a').write(str(f'{my_num}\n'))
        
threads = [threading.Thread(target=scan) for _ in range(500)]

for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

Answer 3

除了threading.Lock之外还有一个重要的标注：

• 使用join使父线程等待分叉线程完成。
• 没有这个，线程仍然会竞争。

假设我在线程完成后使用num ：

import threading

lock, num = threading.Lock(), 0


def operation():
    global num
    print("Operation has started")
    with lock:
        num += 1


threads = [threading.Thread(target=operation) for x in range(10)]
for t in threads:
    t.start()

for t in threads:
    t.join()

print(num)

没有连接，不一致（9 打印一次，否则 10）：

Operation has started
Operation has started
Operation has started
Operation has started
Operation has startedOperation has started

Operation has started
Operation has started
Operation has started
Operation has started9

使用 join，其一致：

Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
10