Dask Distributed Computing Deserialize Error

Question

I have a dask-scheduler and worker running on a linux machine and I trying to send a grid search task from a windows 10 laptop.

from dask.distributed import Client, progress
client = Client('10.133.20.17:8786',processes=False,threads_per_worker=4,n_workers=1, memory_limit='2GB')
from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.externals import joblib
from sklearn.model_selection import GridSearchCV
import pandas as pd

param_grid = {"C": [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0],
          "kernel": ['rbf', 'poly', 'sigmoid'],
          "shrinking": [True, False]}

grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),
                       param_grid=param_grid,
                       return_train_score=False,
                       iid=True,
                       cv=3,
                       n_jobs=-1)

with joblib.parallel_backend('dask'):
    grid_search.fit(X, y)

I am getting this error distributed.protocol.core - CRITICAL - Failed to deserialize

Traceback (most recent call last):
   File "C:\Users\kselvam\SOFTS\anaconda3\lib\site-packages\distributed\protocol\core.py", line 132, in loads
value = _deserialize(head, fs, deserializers=deserializers)
   File "C:\Users\kselvam\SOFTS\anaconda3\lib\site-packages\distributed\protocol\serialize.py", line 183, in deserialize
dumps, loads, wants_context = families[name]
KeyError: None
tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x0000027207EB7950>, <Future finished exception=CancelledError(['_fit_and_score-batch-a4f6fd8cd0354973bb6b49d7f1530390'])>)

I can't understand why it fails to deserialize. When I launch the scheduler as a local cluster (Windows) everything works properly. Does change in the operating system affect the serialization processes ? I have same version of sklearn (0.21.3),tornado (5.1.1) and dask (1.0.0) on linux and windows respectively.

Answer 1

My first guess is that your software isn't uniform across your cluster. You might want to verify your software environment with the following:

client.get_versions(check=True)