VCP Google Cloud Platform's Dataflow Pub/Sub Topic to BigQuery not Pulling data from subscription

Question

I am using GCP to process and store data sent from a simulated IoT device. The data (should) flow in the following path:

[Simulated IoT device with mqtt client] -> [IoT core device] -> [IoT core registry] -> [pub/sub topic] -> [pub/sub subscription] -> [cloud dataflow pub/sub subscription to BigQuery Template] -> [BigQuery Table]

The company I am has some configuration of GCP (it is on a VPC), which requires that I specify a subnet for DataFlow to even run . When dataflow runs after specifying a subnet it doesn't do anything. It doesn't pull data from the subscription and does not output any error messages. It is apparently unaware that data is being sent to the topic and subscription. I have also tried a cloud dataflow pub/sub topic to BigQuery Template and that also does nothing. I have verified that data is being sent to the subscription.

What is the best way to to troubleshoot this issue? I'm at a loss on why there is no communication between these two components.

Answer 1

I was able to finally figure out the issue. The problem was that the region for the virtual private cloud did not match the region for dataflow. I also was not able to use the cloud dataflow pub/sub subscription to BigQuery Template, but I was able to use the cloud dataflow pub/sub topic to BigQuery.

The main takeaway is that the regions matter in this context and if they do not match nothing will happen. If nothing is happening, double check that you have selected the correct regions.