Dask-ML 的 Client() function 中参数的默认值是多少

Question

I am trying to understand Dask-ML's Client() function parameters. Say I have the following code using Dask-ML's Client() function:

from dask.distributed import Client
import joblib

client = Client()

If I don't specify any values for the parameters in the Client() function, what are the default values for the parameters:

(i) n_workers

(ii) threads_per_worker

(iii) memory_limit

From my understanding, Python has the Global Interpreter Lock (GIL) feature which prevents multi-threading. If so, why does Dask-ML's Client() function have the parameter threads_per_worker when multi-threading is prevented in Python?

Does memory_limit refers to the maximum memory limit allowed for each worker/machine/node or does this refer to the maximum memory limit allowed for all combined worker/machine/node?

I have already looked through the documentation in Dask-ML (see here: https://docs.dask.org/en/latest/setup/single-distributed.html ), but the documentation is not clear in regards to these questions above.

Thank you in advance if anyone could explain this?

Answer 1

Calling Client() without any arguments starts a LocalCluster() by default, so

client = Client()

Is really the same as

cluster = LocalCluster()
client = Client(cluster)

So, to start, you might take a look at the LocalCluster documentation.

what are the default values for the parameters:

The ideal values depend both on your hardware, and on your workload. We don't know your workload up-front, but we do know your hardware, and so we try to make sensible decisions based on that.

Today that policy is to split all of your logical cores and memory evenly among the square root of the number of cores that you have. So if you have 12 cores then we would create four processes with three threads each.

This tends to be an ok default in most situations, but we encourage you to play around to see if your workloads perform better under different circumstances.