Increase Paho-MQTT publishers running on Azure Container Instances (Locust Load Test)

Question

We are trying to run a distributed Locust MQTT tests using Azure Container Instances and the Python Paho-MQTT library. We can't run more than 340 clients per worker.

OSError: [Errno 24] Too many open files.

The problem is related with the following issues:

• https://github.com/eclipse/paho.mqtt.python/issues/183
• https://github.com/eclipse/paho.mqtt.python/issues/662

With Docker the soft and hard limits can be changed using --ulimit, but ACI does not accept Docker arguments.

We changed the ACI entry point to increase the open files soft limit running the following bash script:

ulimit -Sn 10000

locust

We added to the locustfile.py:

resource.setrlimit(resource.RLIMIT_NOFILE, (200000, 200000))

We also tried to use the following command:

sudo sysctl -w fs.file-max=500000

sysctl -p

But it returns a permission denied error.

Any idea?

Answer 1

It is not an ACI problem but how the Paho-MQTT client is built. Paho uses the select(..) method which limits us to opening more than 1024 file descriptors.

Each MQTT client means 3 open file descriptors: 3 * 340 = 1020. Over 340 client connections, we hit the 1024 open file descriptors.

We use an MQTT User which inherits from the Paho Client. We overrode the following methods to use the eventfd package.

import eventfd

    [...]
    def loop_start(self) -> Optional[int]:
        [...]
        # self._sockpairR, self._sockpairW = _socketpair_compat()
        self._wake_event = EventFD()
        [...]

    def _reset_sockets(self, sockpair_only: bool = False) -> None:
        [...]
        # if self._sockpairR:
        #   self._sockpairR.close()
        #   self._sockpairR = None
        # if self._sockpairW:
        #   self._sockpairW.close()
        #   self._sockpairW = None
        if self._wake_event:
            self._wake_event = None

    def _packet_queue(
        self,
        command: Literal[48],
        packet: bytearray,
        mid: int,
        qos: int,
        info: Optional[Any] = None,
    ) -> int:
        [...]
        # if self._sockpairW is not None:
        #   try:
        #     self._sockpairW.send(sockpair_data)
        #   except BlockingIOError:
        #     pass
        if self._wake_event is not None:
            with suppress(BlockingIOError):
                self._wake_event.set()
        [...]

    def _loop(self, timeout: float = 1.0) -> int:
        [...]
        # if self._sockpairR is None:
        #   rlist = [self._sock]
        # else:
        #   rlist = [self._sock, self._sockpairR]
        if self._wake_event is None:
            rlist = [self._sock]
        else:
            rlist = [self._sock, self._wake_event]
        [...]
        # if self._sockpairR and self._sockpairR in socklist[0]:
        #   socklist[1].insert(0, self._sock)
        #   try:
        #       self._sockpairR.recv(10000)
        #   except BlockingIOError:
        #       pass
        if self._wake_event and self._wake_event.is_set():
            self._wake_event.clear()
        [...]
    [...]