Docker Services Scaling Container Bottlenecks 249 Containers

Question

AMD 24 CORE Threadripper and 200GB RAM Ubuntu 20 Docker Latest Version Docker Swarm Mode (but the only Host)

I have my docker stack compose file. Scaling the service up I don't have any problems to 249 Containers, but then I have a bottleneck and don't know why Do I need to change Settings somewhere to remove the bottleneck? I already have

fs.inotify.max_queued_events = 100000000
fs.inotify.max_user_instances = 100000000
fs.inotify.max_user_watches = 100000000

in /etc/sysctl.conf as I had a bottleneck with inotify instances by nearly 100 containers, solved that problem with that. But I cant scale past 249 Containers

Answer 1

One issue is certainly going to be IP availability if this is docker swarm, as overlay networks - the default on swarm - are automatically /24 and thus limited to ~255 hosts.

So:
a. attach the service to a manually created network that you have scoped to be larger than /24 b. ensure that the services endpoint mode is dnsrr as vip's cannot be used safely with networks with larger than /24 address space.