Is there a way to create a batch job in dataflow which reads from a pubsub topic?

Question

I have a dataflow pipeline in streaming mode as it reads from a pubsub topic. I wish to modify it to batch mode. I need so inorder to use FlexRS as it is supported only for batch mode. Beam version : 2.13.0

Answer 1

You can look into using BoundedReadFromUnbounded source that allows to convert unbounded collection to bounded.

UnboundedSource<String> unboundedSource  = ...;
PCollection<String> boundedCollection=
     p.apply(Read.from(unboundedSource).withMaxNumRecords(10));

You can also utilize withMaxReadTime