In BigQuery SQL, how to track resource usage to avoid "Resources exceeded during query execution" error

Question

In BigQuery, Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex. is an error we receive when our queries are too big or complex.

For queries that run successfully, how can we see how close we are to receiving this error? What is the criteria for resources here, and what are our limits? We have a few large queries that may be pushing close to this error, but we don't want to unexpectedly receive the error one day as our source table for this query continues to grow.

It is easy for us to find bytes queried, as the editor makes it quite clear. And after executing a query, we can see the following in Execution Details

Is one of either Slot time consumed or Bytes shuffled the relevant metric here, with their being some limit after which the Resources exceeded error is thrown? Or is there some other criteria that determines when this error is thrown?

For added context, none of our tables query a particularly large amount of data, our largest query is under 20GB processed, however our larger queries do have a lot of sub-queries with t1 as (), t2 as (), t3 as (), t4 as (), t5 as (), ... , where the subqueries also reference the previous sub-queries, and I believe this is what is leading to the resources exceeded errors.

Edit: Output from our largest query, the one that prompted this posting. I refactored 2 of the subqueries into their own CTEs, so that this big query run would be successful. This obviously still seems large...

Answer 1

We've encountered this error quite a bit. According to this article: https://medium.com/@jamiekt/what-to-do-about-bigquery-error-resources-exceeded-during-query-execution-e80734b8c9b6 BigQuery computes a complexity score and will output an error if it crosses a threshold.

At this time, there is no easy way to know the complexity score of a query and how close to the limit it is.

However, if a complex query is working, that same query should keep working even if the underlying data continuously increases in size. Or rather, it shouldn't fail with a "Query is too complex" error as the complexity score would remain unchanged, but could end up failing with other errors, such as "The query could not be executed in the allotted memory".

To help alleviate the "Query is too complex" it is recommended to use temp tables ( https://cloud.google.com/bigquery/docs/writing-results ), avoid UNIONs when possible, reducing the amount of CTEs and nested views.