什么是 BigQuery DML 配额限制

Question

I was under impression that BigQuery DML has no longer INERT restriction, however, one of my ingestion workflows that runs INSERT DML uniformly distributed across the day, on average 80 DML every 2 min (~35-70B records transformations and aggregation per day) failed occasionally.

What's strange I am seeing only a few INSERT DML queries failed and only for some hours. When looking at pending jobs, around when errors took place there are no more than a few queries job in the pending state. All DML queries use the reservation.

I wonder what is and how the quota is computed, besides that documentation claims there is no DML INSERT quota.

Here is the error I am seeing. {"location":"max_dml_outstanding_per_table","message":"Quota exceeded: Your table exceeded quota for total number of dml jobs writing to a table, pending + running. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors","reason":"quotaExceeded"}

To better understand my case, I am using bqtail ingestion workflow defined as:

When:
  Prefix: "/xxxxx/xxxx/"
  Suffix: ".gz"
Async: true
Batch:
  MultiPath: true
  Window:
    DurationInSec: 120

Dest:
  Pattern: '.+/(\d{4})/(\d{2})/(\d{2})/.+'
  Table: myproject.selector.selection_$Mod(80)_$1$2$3
  SourceFormat: NEWLINE_DELIMITED_JSON
  Transient:
    Dataset: temp
    Balancer:
      MaxLoadJobs: 100
      ProjectIDs:
        - myproject-transient1
        - myproject-transient2
        - myproject-transient3
        - myproject-transient4
        - myproject-transient5
  Schema:
    Template: myproject.selector.selection_tmpl
  SchemaUpdateOptions:
    - ALLOW_FIELD_ADDITION
  WriteDisposition: WRITE_APPEND

OnSuccess:
  - Action: query
    Request:
      SQL: INSERT INTO `myproject.selector.xxx_agg1`( ....) SELECT ... FROM $TempTable GROUP BY x, y, z
    OnSuccess:
      - Action: query
        Request:
          SQL: INSERT INTO `myproject.selector.xxx_agg2`( ....) SELECT ... FROM $TempTable GROUP BY x1, y1, z1
        OnSuccess:
          - Action: delete

In summary: every two minutes data files are batched upto 80 load requests into transient tables and then copied to the final destination tables: myproject.selector.selection_$Mod(80)_$1$2$3 (80 different tables suffixed by date).

Ingestion work is distributed between 5 transient projects, the typical batch would be up to 1M records, each load job taking around 22 sec and copy job taking 1 sec. After a successful copy, the first DML executes, then the second.

Answer 1

As per GCP support team. The error message indicated that DML jobs were hitting DML_ALL_JOBS_CONCURRENT limit.

DML_ALL_JOBS_CONCURRENT is not a numbered limit itself, it just gets triggered when either the INSERT concurrency limit or the UPDATE/DELETE/MERGE concurrency limit has been reached.

DML INSERT allows up to 100 concurrent jobs (pending+running) per table. DML MERGE/UPDATE/DELTE up 20 concurrent jobs (pending+running) per table.

Answer 2

This quota error means you're submitting jobs faster than they can be finished by BigQuery. BigQuery can only run a certain number of DML jobs concurrently on a table (running jobs). When jobs are received after this limit is exceeded, the jobs are put into a queue to wait for execution (pending jobs). When limit of this queue is also exceeded, you receive this quota error, "Your table exceeded quota for total number of dml jobs writing to a table, pending + running".

Retry the submission of these jobs, with exponential backoff, should help in this situation.

The streaming API is another option for frequent, small appends to the table. It allows for a much higher QPS.

Answer 3

Thanks to your comment, you should reach one limit. When you look at the quotas documentation for DML you can read this

BigQuery DML statements have no quota limits.

However, DML statements are counted toward the maximum number of table operations per day and partition modifications per day. DML statements will not fail due to these limits.

In addition, DML statements are subject to the maximum rate of table metadata update operations. If you exceed this limit, retry the operation using exponential backoff between retries.

If you follow the latest link, on the table metadata update operation , you have this

Maximum rate of table metadata update operations — 5 operations every 10 seconds per table

The table metadata update limit includes all metadata update operations performed by using the Cloud Console, the classic BigQuery web UI, the bq command-line tool, the client libraries, by calling the tables.insert, tables.patch, or tables.update API methods, or executing ALTER TABLE DDL statements. This limit also applies to job output.

---

So, in summary, you perform more than 5 tables insert tables.insert over 10 secondes and you have an issue with an retry exponential backoff.

Sometime the quota is tolerant, sometime it reject you. This can depend on the global platform status.

Try to insert is only 1 request in the same table (you can use temp table and then perform a global insert query over all these temp tables)