How to export data from BigQuery and store it as .csv in Google Storage

Question

How do i extract the data from the table using the pipeline and store it as csv in the GS? So far i've only been able to extract the data in a simple text format by extracting each field, concatenating it to a string and then outputting it.

Does anyone know a method for this? Thanks.

Answer 1

Reading from Bigquery using - BigQuery I/O

To read from a BigQuery table, you apply a BigQueryIO.Read transform. BigQueryIO.Read returns a PCollection of BigQuery TableRow objects, where each element in the PCollection represents a single row in the table.

You can read an entire BigQuery table by supplying the BigQuery table name to > BigQueryIO.Read by using the .from operation. The following example code shows > how to apply the BigQueryIO.Read transform to read an entire BigQuery table:

PipelineOptions options = PipelineOptionsFactory.create(); Pipeline p = Pipeline.create(options);

PCollection weatherData = p.apply( BigQueryIO.Read .named("ReadWeatherStations") .from("clouddataflow-readonly:samples.weather_stations"));

Reading from BigQuery

Writting to CSV - using - TextIO.Write

To output data to text files, apply TextIO.Write to the PCollection that you want to output. Keep the following things in mind when using TextIO.Write:

You may only apply TextIO.Write to a PCollection. You may need to use a simple ParDo to format your data from an intermediate PCollection to a PCollection prior to writing with TextIO.Write. Each element in the output PCollection will represent one line in the resulting text file. Dataflow's file-based write operations, like TextIO.Write, write to multiple output files by default. See Writing Output Data for more information.

PCollection filteredWords = ...; filteredWords.apply(TextIO.Write.named("WriteMyFile") .to("gs://some/outputData"));

Writing to Text Files