简体   繁体   English

使用java将json数据添加到Bigquery中,而无需使用作业来加载数据

[英]Strreaming json Data into Bigquery using java.With out using a job to load data

I am new to bigquery so i do not fully understand how to stream data to bigquery, this is my problem, I have jsonInString, mapped from object like this 我是bigquery的新手,所以我不完全了解如何将数据流传输到bigquery,这是我的问题,我有jsonInString,是从这样的对象映射的

    String customerJsonInString = mapper.writeValueAsString(customer);

   {
  "id": "1",
  "first_name": "John",
  "last_name": "Doe",
  "dob": "1968-01-22",
  "addresses": [
    {
      "status": "current",
      "address": "123 First Avenue",
      "city": "Seattle",
      "state": "WA",
      "zip": "11111",
      "numberOfYears": "1"
    },
    {
      "status": "previous",
      "address": "456 Main Street",
      "city": "Portland",
      "state": "OR",
      "zip": "22222",
      "numberOfYears": "5"
    }
  ]
}

the table has been created with the correct schema. 该表已使用正确的架构创建。 now i want to stream this data into bigquery (insert row),i am using the example that is on ( https://cloud.google.com/bigquery/streaming-data-into-bigquery#bigquery-stream-data-java ) to adapt it to mine, this is what i tried, 现在我想将此数据流式传输到bigquery(插入行)中,我使用的是以下示例( https://cloud.google.com/bigquery/streaming-data-into-bigquery#bigquery-stream-data-java )以使其适合我的,这就是我尝试过的,

TableId tableId = TableId.of(DATASET_NAME,TABLE_NAME);
        Map<String, Object> recordsContent = new HashMap<>();
        recordsContent.put("Customer", customerJsonInString);
        InsertAllResponse response = bigquery.insertAll(InsertAllRequest.newBuilder(tableId)
                        .addRow("rowId", recordsContent)
                        .build());
        if (response.hasErrors()) {
            for (Entry<Long, List<BigQueryError>> entry : response.getInsertErrors().entrySet()) {
            }
        }

I'm guessing the schema of the table is id, first_name, last_name, ... instead of customer ? 我猜该表的架构是id, first_name, last_name, ...而不是customer In that case you'll need to set the fields one by one instead of as a whole json string, eg recordsContent.put("id", 1) . 在这种情况下,您需要一个一个地设置字段,而不是整个json字符串,例如recordsContent.put("id", 1)
Check the example with comments . 检查带有注释示例

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM