Dask shared futures with Channels and memory usage

Question

I'm playing with the dask functionality described here " Shared Futures With Channels ".

As explained in the document i managed to submit futures and add them to the channel. Then from another client I manage to run through the future and to get the result from them.

In my use case, it's possible to have a lot of futures loaded to the channel. When i look at the scheduler i see that the memory usage growth even and stay high even after that i get the result.

Is there a way to unload the futures from a channel ?

Answer 1

As of January 2017 there is no mechanism to clear out specific futures from a channel, though this could possibly be added. You might want to raise an issue here .

You might also consider creating a channel with a maximum length using the maxlen= parameter

chan = Channel('my-channel', maxlen=100)

You will need to do this the first time you create the channel.