使用 go 更新/覆盖而不是插入插入 bigquery

Question

I have an aws lambda written in Go that should insert into bigquery. It reference the cloud.google.com/go/bigquery package.

client, err := bigquery.NewClient(ctx, projectID, gcpOption)
if err != nil {
    println(fmt.Sprintf("error creating new client, %v", err))
    return fmt.Errorf("bigquery.NewClient: %v", err)
}
defer client.Close()

inserter := client.Dataset(datasetID).Table(tableID).Inserter()
if err := inserter.Put(ctx, items); err != nil {
    println(fmt.Sprintf("error inserting, %v", err))
    if multiError, ok := err.(bigquery.PutMultiError); ok {
        for _, err1 := range multiError {
            for _, err2 := range err1.Errors {
                fmt.Println(err2)
            }
        }
    } else {
        fmt.Println(err)
    }
    return err
} else {
    println("Inserted record")
}

When run, a record will be inserted, but running again will result in the previously inserted row being updated. This is not the behaviour I was expecting. I am relatively new to Golang and GCP, so perhaps I have the wrong expectations?

The table in big query is not partitioned. Items is an array of structs.

Answer 1

The Inserter can be used to achieve at-least-once data insertion semantics. The insert mechanism is not capable of upsert behavior, which is what you appear to be describing.

It's unclear to me how you're validating this behavior, but I'd take another look at that as a starting point.

More information about the tabledata.insertAll streaming API which underlies the go Inserter can be found here: https://cloud.google.com/bigquery/streaming-data-into-bigquery