Docker Mac network_mode host and kubernetes using kind

Question

Saw on a few issues (such as https://github.com/docker/for-mac/issues/2716 ) that network_mode: host is not supported on Mac.

However, when I was trying to run a Kubernetes cluster using kind , and run Spark on K8s, using the Spark master address as k8s://127.0.0.1:59369 which is the same address and port as the K8s control plane, I get a Connection Refused error unless I am using network_mode: host . Error line:

Caused by: java.net.ConnectException: Failed to connect to /127.0.0.1:59369

This confuses me as I thought network_mode: host should have no effect on Mac, yet there is an effect, and it is blocking me from using bridge network and adding port mapping for another application in this container.

Any ideas how to resolve this?

Answer 1

Specifying host networking on Docker Desktop setups does something; the problem is that the "host" network it specifies isn't actually the "host" you'd intuitively expect as someone typing on the keyboard.

Docker Desktop on MacOS launches a hidden Linux VM, and the containers run inside that VM. When you run docker run -p , Docker Desktop is able to forward that port from the VM to the host, so port mappings work normally. If you use docker run --net=host , though, you get access to the VM's host network; since this generally disables Docker's networking layer, Docker isn't able to discover what the container might be doing, and it can't forward anything from the actual host into the VM to the container. That's the way host networking "doesn't work".

In practice I see host networking suggested for four things:

1. Processes with unpredictable or a very large number of port mappings, where docker run -p can't be used
2. To actually manage the host's network environment, in spite of running in a container
3. To access non-container processes on the same host, where host.docker.internal would also work
4. To access the published ports of other containers, without setting up Docker networking

I think here's you're running into this last case. Kind is publishing a port for the Kubernetes API service, so port 59369 is accessible on both the actual host and in the Linux VM. Now if your Spark container activates host networking it is using the VM's host network, but the other container's published port is still accessible there, which is why the http://localhost:59369 URL still works.