GCP 中的物联网管道

Question

I have an IoT Pipeline in GCP that is structured like:

IoT Core -> Pub/Sub -> Dataflow -> BigQuery

I am using esp32 devices to send data with new data being sent every 2 seconds. For now I am testing with only 4 devices but ultimatly the project will consist of hundreds of esp32 devices each sending data after every 2 seconds. The issue is that even with 4 devices the unacked message count in Subscription goes up to 1260 messages. Even though these messages are not lost they are simply just delayed, this could end up causing issues when I have to use hundreds of devices. So I need to alter my pipeline such that the data can be stored successfully without such a delay. The data sent is in csv format. It is converted to JSON in Dataflow using a Javascript UDF, then uploaded to Bigquery using the google defined templates: Pub/Sub to BigQuery . All devices are using the same Pub/Sub topic and subscription. Data from all devices is uploaded into the same BigQuery table. If it helps then it is also possible to store data somewhere else like in Cloud Storage (if that is faster) first and then upload all the data to BigQuery later on (after every hour or something) but ultimately I require all my data to be inside BigQuery. Please suggest the how I can improve my pipeline.

Answer 1

This error was being caused because after every 10 seconds the pub/sub resent the messages that had not yet been acknowledged. This caused the total number of messages to grow rapidly as the number of devices sending the messages and the rate at which they sent them was already very high. So I increased this wait time to 30 seconds and the system calmed down. Now there is no large group of unacknowledged messages forming when I run the pipeline.