Google Cloud Platform: accumulate data from Pub/Sub to files in Cloud Storage without Dataflow

Question

I'm trying to figure out if there is a service on GCP which would allow consuming a stream from Pub/Sub and dump/batch accumulated data to files in Cloud Storage (eg every X minutes). I know that this can be implemented with Dataflow, but looking for more "out of the box" solution, if any exists.

As an example, this is something one can do with AWS Kinesis Firehose - purely on configuration level - one can tell AWS to dump whatever is accumulated in the stream to files on S3, periodically, or when accumulated data reaches some size.

The reason for this is that - when no stream processing is required, but only need to accumulate data - I would like to minimize additional costs of:

• building a custom piece of software, even a simple one, if it can be avoided completely
• consuming additional compute resources to execute it

To avoid confusion - I'm not looking for a free of charge solution, but the optimal one.

Answer 1

Google maintains a set of templates for Dataflow to perform common tasks between their services.

You can use the "Pubsub to Cloud Storage" template by simply plugging in a few config values - https://cloud.google.com/dataflow/docs/templates/provided-templates#cloudpubsubtogcstext