在目标表架构中使用 BIGDECIMAL 时无法设置对 BigQuery 的 GCP Pub/sub 订阅

Question

Objective : Setup a GCP Pub/sub subscription:

• on a topic associated to a protobuf schema
• which directly routes the JSON-encoded messages to a BigQuery table containing a BIGNUMERIC field

Issue : Creating the topic consistently errors out with a generic API returned error: 'Request contains an invalid argument.' .

What was attempted : After many combinations of protobuf schema + target table schema, the conversion of any protobuf numeric type to a BigQuery BIGNUMERIC field seems to be the offending aspect.

I found this official doc table which describes the available conversions, and I notice that the target ZetaSQL types do not feature BIGNUMERIC at all.

Question : Am I missing something, or does it actually mean there is no way to do Pub/sub-to-BigQuery ingestion if the target table has a BIGNUMERIC field?

Answer 1

The BIGNUMERIC type is not currently supported in Pub/Sub to BigQuery subscriptions. The only way to ingest into BigQuery from Pub/Sub with this datatype would be to use Dataflow.