Reading streaming data using Google Cloud Dataflow SDK

Question

I am working on an IOT application where I need to read streaming data from a PubSub topic. I want to read this data using Google Cloud Dataflow SDK. I am using Java 1.8

I am using the trial version of Google Cloud Platform. When I was using PubSubIO.Read method to read the streaming data, I was getting errors in the log file that my project does not have enough CPU quota to run the app.

So I want to read the streaming data using Google Cloud Dataflow SDK.

Can someone please let me know where can I find complete examples of reading the streaming data using Google Cloud Dataflow SDK.

Thanks in advance.

Answer 1

A number of complete examples are available in Cloud Dataflow documentation under Complete Examples and, linked from there, on Github, also under Complete Examples .

According to your error message, you indeed have not enough CPU quota to run the default 3x 4-threaded ( n1-standard-4 ) workers. The CPU quota for Google Cloud trial is 8 CPUs.

You can configure your job to require fewer CPUs, eg by using fewer workers (eg --numWorkers=1 ) or a different machine type ( --workerMachineType=n1-standard-1 )