Azure IoT Hub: What happens when sending data if daily message limit is exceeded?

Question

We are using Azure's IoT Hub as backend for our IoT solution. We noticed that we had an unexpected spike in message amount during one day, and we exceeded our daily limit of 400k messages.

From client application's logs we couldn't find anything out of the normal: based on the logs it seems that client kept sending data despite the daily limit being exceeded.

So my question is: what happens when client application sends data using DeviceClient.SendEventBatchAsync when daily message limit is already exceeded? Are the messages just dropped cold blooded despite there was no exceptions? Or is there happening something that I have failed to catch? We are using C# .NET Microsoft.Azure.Devices.Client version 1.19.0 and sending the data using MQTT protocol.

Answer 1

To accommodate burst traffic, IoT Hub accepts requests above the throttle for a limited time. The first few of these requests are processed immediately. However, if the number of requests continues violate the throttle, IoT Hub starts placing the requests in a queue and processed at the limit rate. This effect is called traffic shaping. Furthermore, the size of this queue is limited. If the throttle violation continues, eventually the queue fills up, and IoT Hub starts rejecting requests with 429 ThrottlingException.

For example, you use a simulated device to send 200 device-to-cloud messages per second to your S1 IoT Hub (which has a limit of 100/sec D2C sends). For the first minute or two, the messages are processed immediately. However, since the device continues to send more messages than the throttle limit, IoT Hub begins to only process 100 messages per second and puts the rest in a queue. You start noticing increased latency. Eventually, you start getting 429 ThrottlingException as the queue fills up, and the "number of throttle errors" in IoT Hub's metrics starts increasing.

According to the docs

So yea, it gets batched until it start throwing exceptions when the queue is full. You should reduce the number of messages and consider choosing a MQTT library that supports client side batching in case there's burst data.

Answer 2

Your MQTT device should be disconnected and the Send and Receive operations are blocked for this hub until the next UTC day .

In the case of using the https protocol, the following response is sent by Azure IOT Hub:

{
  "Message": "{\"errorCode\":403002,\"trackingId\":\"c41eb2a0f7764132aa31a7f3ff97a1ce-G:3-TimeStamp:06/20/2019 12:36:43\",\"message\":\"Total number of messages on IotHub 'xxxxxxxxx' exceeded the allocated quota. Max allowed message count : '8000', current message count : '8448'. Send and Receive operations are blocked for this hub until the next UTC day. Consider increasing the units for this hub to increase the quota.\",\"timestampUtc\":\"2019-06-20T12:36:43.5570129Z\"}",
  "ExceptionMessage": ""
}

That's for the F1 scale tier and it should be the same behavior for any scale tier when the daily allocated Send/Receive message operations has been exceeded.