dask 执行卡在 LocalCluster

Question

I am using an EC2 VM with 16 cores and 64GB ram . Wrote a Dask program that applies a filter on a dataframe and does a concat with another one and then writes the data back to disk. If I run it in LocalCluster mode by calling simply client = Client() , the execution gets stuck at some point after writing some data. During this period, the CPU utilisation is very very low and I can easily understand that nothing is getting executed. Also size of the part files stops increasing at this point. This goes on forever. But If I execute it without creating LocalCluster , it runs very slowly (low CPU utilisation) and finishes up the program. Trying to understand how I can fix this.

Note: Nobody else is using the VM and the data size ranges from 3GB to 25GB.

Dask version: 2.15.0 & 2.17.2

Answer 1

Unfortunately there is not enough information in your question to provide a helpful answer. There are many things that could be going on.

In this situation we recommend watching the Dask dashboard, which can give you much more information about what is going on. Hopefully that can help you identify your issue.

https://docs.dask.org/en/latest/diagnostics-distributed.html