Python ThreadPoolExecutor (concurrent.futures) 内存泄漏

Question

你好我正在尝试加载一个大列表== list.txt并将其发送到 Function== Do_something() with concurrent.futures.ThreadPoolExecutor问题是无论我做什么，内存都会变重，起初我认为原因是我list.txt打开为一个变量（列表），因此我将代码更改为for i in open("list.txt").readlines() from list = open("list.txt").readlines()但问题仍然存在，完成工作后是否可能逐行清除内存？

我的代码：

import time
from concurrent.futures import ThreadPoolExecutor



def Do_something(i):
    
    time.sleep(5) #Do Something ~ take few sec 
    
    pass


if __name__ == "__main__":
    #list = open("list.txt").readlines()
    #even with 1 thread code have problem
    with ThreadPoolExecutor(1) as executor:
        try:
            #list.txt == 10,000,000 Line
            [executor.submit(Do_something , i )for i in open("list.txt").readlines()]
            
        except Exception as exx:
            pass

Answer 1

首先，完全删除.readlines()调用； 文件对象已经是它们行的可迭代对象，所以你所做的就是强制它创建一个包含所有行的list ，然后是使用这些行调度的所有任务的另一个list 。 通常， .readlines()从来都不是必需的（它只是list(fileobj)的微优化，当您不需要list时，您不想使用它）。

其次，在从任何任务中获取结果之前，您明确尝试预先为所有输入行创建任务。 虽然避免.readlines()可以节省包装所有这些行的list的开销，但您仍然试图将它们全部保存在内存中，每个任务一个。 如果您没有足够的 RAM 来同时处理所有任务，则无法执行此操作。

如果你想排队一定数量的任务，在它们完成时处理结果并排队新任务，你可以做这样的事情（改编自补丁以使Executor.map避免你遇到的问题）：

import collections
import itertools
import time


def executor_map(executor, fn, *iterables, timeout=None, chunksize=1, prefetch=None):
    """Returns an iterator equivalent to map(fn, iter).
    Args:
        executor: An Executor to submit the tasks to
        fn: A callable that will take as many arguments as there are
            passed iterables.
        timeout: The maximum number of seconds to wait. If None, then there
            is no limit on the wait time.
        chunksize: The size of the chunks the iterable will be broken into
            before being passed to a child process. This argument is only
            used by ProcessPoolExecutor; it is ignored by
            ThreadPoolExecutor.
        prefetch: The number of chunks to queue beyond the number of
            workers on the executor. If None, a reasonable default is used.
    Returns:
        An iterator equivalent to: map(func, *iterables) but the calls may
        be evaluated out-of-order.
    Raises:
        TimeoutError: If the entire result iterator could not be generated
            before the given timeout.
        Exception: If fn(*args) raises for any values.
    """
    if timeout is not None:
        end_time = timeout + time.monotonic()
    if prefetch is None:
        prefetch = executor._max_workers
    if prefetch < 0:
        raise ValueError("prefetch count may not be negative")

    argsiter = zip(*iterables)
    initialargs = itertools.islice(argsiter, executor._max_workers + prefetch)

    fs = collections.deque(executor.submit(fn, *args) for args in initialargs)

    # Yield must be hidden in closure so that the futures are submitted
    # before the first iterator value is required.
    def result_iterator():
        nonlocal argsiter
        try:
            while fs:
                if timeout is None:
                    res = fs.popleft().result()
                else:
                    res = fs.popleft().result(end_time - time.monotonic())

                # Dispatch next task before yielding to keep
                # pipeline full
                if argsiter:
                    try:
                        args = next(argsiter)
                    except StopIteration:
                        argsiter = None
                    else:
                        fs.append(executor.submit(fn, *args))

                yield res
        finally:
            for future in fs:
                future.cancel()
    return result_iterator()

获得该map实用程序后，您可以将代码更改为：

if __name__ == "__main__":
    with ThreadPoolExecutor() as executor:
        try:
            #list.txt == 10,000,000 Line
            with open("list.txt") as f:  # Use with statements to get deterministic file close
                for res in executor_map(executor, Do_something, f):
                    pass  # If Do_something returns useful values, you can use them here
                          # with each result going into res

        except Exception as exx:
            pass

一次只存在有限数量的任务（多于工作人员的数量，但有些可能已经有你没有提取的结果），文件被延迟读取，所以它不会破坏你的 RAM。