I am trying to modify this example as follows: Above gives me following error: How can I fix this? ...
I am trying to modify this example as follows: Above gives me following error: How can I fix this? ...
I'm using Beam (and Scio, though feel free to answer this question for PCollections too) to read from multiple tables in BigQuery. Because I'm reading ...
I'm implementing a set of pipeline in dataflow I'm looking for why do you choose scala instead of apache beam? Why you chose between one or another? ...
I'm using Apache Beam 2.28.0 on Google Cloud DataFlow (with Scio SDK). I have a large input PCollection (bounded) and I want to limit / sample it to a ...
I want to know about, is it possible to Fetch a Spotify music list without an Access token or Aothuentication in mobile flutter SDK or web API. becaus ...
I am running a streaming beam job on a flink cluster where I am getting the following exception. The streaming job is getting data from the apache ...
I have been running an apache beam based data ingestion job which parses an input CSV file and writes data on a sink. This job works fine when one job ...
I'm trying to use Apache Beam (via Scio) to run a continuous aggregation of the last 3 days of data (processing time) from a streaming source and outp ...
I'm trying to aggregate (per key) a streaming data source in Apache Beam (via Scio) using a stateful DoFn (using @ProcessElement with @StateId ValueSt ...
I want to convert SCollection[String] to Seq[String] or List[String]. For example, I have a variable called ids. When I save it to Cloud Storage, ...
I'm trying to use Beam's stateful processing on Dataflow, but I get these errors in the log every time I try to output data. The result is that nothin ...
I try to get a simple SCIO code running. Trying to use Foo in a SCollection it leads to an error: There is a lot written in the error message. I ...
Is there anyway to trigger early output of windows when running in batch mode? I've tried a number of triggers with the Dataflow runner to get early w ...
Using the AfterPane.elementCountAtLeast trigger does not work when run using the Dataflow runner, but works correctly when run locally. When run on Da ...
Is there any way to view the contents of an SCollection when running a unit test (PipelineSpec)? When running something in production on many machin ...
I want to do parameterized tests with SCIO JobTest and Scala Test. I use TableDrivenPropertyChecks that allows, via a a forAll to do parameterized tes ...
I have a problem. I created a SCIO (Apache Beam) project via a sbt archetype : sbt new spotify/scio.g8 The goal of this Job is to Read a parquet fil ...
I am using Spotify's Scio library for writing apache beam pipelines in scala. I want to search for files under a directory in a recursive way on a fil ...
I am trying to run my first Scio pipeline on Dataflow . The code in question can be found here. However I do not think that is too important. My firs ...
I have a pipeline with a set of PTransforms and my method is getting very long. I'd like to write my DoFns and my composite transforms in a separate ...