如果我在 HPC 上使用 Dask-Jobqueue，是否还需要使用 Dask-ML 来运行 scikit-learn 代码？

Question

If I am using Dask-Jobqueue on a High Performing Computer (HPC), do I still need to use Dask-ML (ie. joblib.parallel_backend('dask' ) to run scikit-learn codes?

Say I have the following code:

from dask_jobqueue import PBSCluster

cluster = PBSCluster(cores=36,   
                     memory='100GB',   
                     project='P48500028',   
                     queue='premium',   
                     interface='ib0',
                     walltime='02:00:00')

cluster.scale(100)  
                   
from dask.distributed import Client
client = Client(cluster)   


from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

X, y = make_classification(n_samples=2000, n_features=20, n_classes=2, random_state=0)

param_grid = {"C": [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0],
              "kernel": ['rbf', 'poly', 'sigmoid'],
              "shrinking": [True, False]}

grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),
                           param_grid=param_grid,
                           return_train_score=False,
                           iid=True,
                           cv=3,
                           n_jobs=-1)


import joblib

with joblib.parallel_backend('dask'):
    grid_search.fit(X, y)

Since I am using Dask-Jobqueue on a HPC (ie. I am connected to an instance of the HPC), when I run my code would all of my code be distributed to a cluster already (since I have specified cluster.scale(100) )? If yes, then do I still need the last 3 lines of code above which uses Dask-ML? Or can my code be like this:

from dask_jobqueue import PBSCluster

cluster = PBSCluster(cores=36,   
                     memory='100GB',   
                     project='P48500028',   
                     queue='premium',   
                     interface='ib0',
                     walltime='02:00:00')

cluster.scale(100)  
                   
from dask.distributed import Client
client = Client(cluster)   


from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

X, y = make_classification(n_samples=2000, n_features=20, n_classes=2, random_state=0)

param_grid = {"C": [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0],
              "kernel": ['rbf', 'poly', 'sigmoid'],
              "shrinking": [True, False]}

grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),
                           param_grid=param_grid,
                           return_train_score=False,
                           iid=True,
                           cv=3,
                           n_jobs=-1)

grid_search.fit(X, y)

Will the last line of code above grid_search.fit(X, y) not run on any Dask cluster since I have removed joblib.parallel_backend('dask') ? Or will it still run on a cluster since I have earlier on declared cluster.scale(100) ?

Many thanks in advance.

Answer 1

Will the last line of code above grid_search.fit(X, y) not run on any Dask cluster since I have removed joblib.parallel_backend('dask')?

Correct. Scikit-Learn needs to be told to use Dask

Or will it still run on a cluster since I have earlier on declared cluster.scale(100)?

No. Dask is unable to automatically parallelize your code. You need to either tell Scikit-Learn to use Dask with the joblib decorator, or else use the dask_ml GridSearchCV equivalent object.