从数据流向BigQuery插入数据

Question

Previously, had PCollection formattedResults; and I was using below code to insert rows in big query:

                   // OPTION 1
PCollection<TableRow> formattedResults = ....
formattedResults.apply(BigQueryIO.Write.named("Write").to(tableName)
                            .withSchema(tableSchema)
                            .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
                            .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND));

And all rows was directly inserted in BigQuery, all well till here. But now I have started to dynamically identify the table name and its row so am creating PCollection as below: (String will be the table name and then its row as value)

PCollection<KV<String, TableRow>>   tableRowMap // OPTION 2

Also, I am creating group of rows which will go in same table as:

PCollection<KV<String, Iterable<TableRow>>> groupedRows  //OPTION 3

where key (String) is the BQ table name and value is the list of rows to be inserted in BQ.

With option 1, I was able to easily insert rows in BQ using code shown above but same code cannot be used with OPTION 2 or OPTION 3 because in this case my table name is key in map. Is there a way to insert rows in table using OPTION 2 or OPTION 3. Any link or code sample will be great help.

Answer 1

The closest thing that Dataflow is writing to a table per window (and you can create your own BoundedWindow subclass and WindowFn to include whatever data you want in the window). To do this, use

to(SerializableFunction<BoundedWindow,String> tableSpecFunction)

on BigQueryIO.Write.

Note that this functionality uses BigQuery's streaming upload feature, which is limited to 100MB/s per table. Additionally, uploads are not atomic, so a failed batch job may upload only part of the output.

Answer 2

You've also got the option to create you're own DoFn that directly inserts data into bigquery, instead of relying on BigQueryIO.Write. Technically you need to create a BigQueryTableInserter , you can use the insertAll(TableReference ref, List<TableRow> rowList) to insert stuff into your desired table.

You can create a TableReference using something like: new TableReference().setProjectId("projectfoo").setDatasetId("datasetfoo").setTableId("tablefoo")

This isn't 100% recommended as BigQueryIO does some nice stuff to split up the rows that need inserting to maximise throughput and handles retries properly.