长时间运行的进程上的 ThreadPoolExecutor

Question

I want to use ThreadPoolExecutor on a webapp (django),

All examples that I saw are using the thread pool like that:

with ThreadPoolExecutor(max_workers=1) as executor: code I tried to store the thread pool as a class member of a class and to use map fucntion

but I got memory leak, the only way I could use it is by the with notation

so I have 2 questions:

1. Each time I run with ThreadPoolExecutor does it creates threads again and then release them, in other word is this operation is expensive?

2. If I avoid using with how can I release the memory of the threads

thanks

Answer 1

Normally, web applications are stateless. That means every object you create should live in a request and die at the end of the request. That includes your ThreadPoolExecutor . Having an executor at the application level may work, but it will be embedded into your web application instead of running as a separate group of processes.

So if you want to take the workers down or restart them, your web app will have to restart as well.

And there will be stability concerns, since there is no main process watching over child processes detecting which one has gotten stale, so requires a lot of code to get multiprocessing right.

Alternatively, If you want a persistent group of processes to listen to a job queue and run your tasks, there are several projects that do that for you. All you need to do is to set up a server that takes care of queueing and locking such as redis or rabbitmq, then point your project at that server and start the workers. Some projects even let you use the database as a job queue backend.