OSError: Timed out trying to connect to Dask Scheduler from GCE to GKE - externalIP

Question

I'm trying connect my dask Client to the dask scheduler service deployed on my GKE. The deployment has been done with Helm and the dask-scheduler service has LoadBalancer as type, so it has an externalIP.

When trying to connect from inside the cluster, with Jupyter it works. When connecting from a GCE, with

Client('tcp://<scheduler-externalIP>:<scheduler-port>')

I get a timeout error:

`Traceback (most recent call last):
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 319, in connect
handshake = await asyncio.wait_for(comm.read(), time_left())
File "/usr/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "parsing.py", line 25, in <module>
client = Client('tcp://34.147.64.50:8786')
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/client.py", line 763, in __init__
self.start(timeout=timeout)
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/client.py", line 945, in start
sync(self.loop, self._start, **kwargs)
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/utils.py", line 327, in sync
raise exc.with_traceback(tb)
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/utils.py", line 310, in f
result[0] = yield future
File "/home/<user>/.local/lib/python3.7/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/client.py", line 1035, in _start
await self._ensure_connected(timeout=timeout)
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/client.py", line 1096, in _ensure_connected
self.scheduler.address, timeout=timeout, **self.connection_args
File "/home/<user>/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 326, in connect
) from exc
OSError: Timed out during handshake while connecting to tcp://<externalIP>:8786 after 30 s`

Answer 1

First thing to check is the GCP's firewall - you may need to create a rule to allow such connections.

In this case port 8787 is used and Dask uses regular TCP connections.

So - this may for example look like this: gcloud compute firewall-rules create allow-dask-connections --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:8787 --target-tags=your-cluster-node-network-tag Make sure firewall lets the traffic in since by default any ports other than 22 and 3389 are blocked.

Other than that it's not much from the GCP's side that can block the traffic.