[英]BigQuery BI Engine: how to choose a good reservation size?
我正在启用 BI Engine 来加快我的查询速度并为我在欧盟地区的项目节省成本。
设置预留大小的最佳选择是什么? 1GB、2GB、4GB?
我如何做出这个决定?
下面是一个 SQL 脚本,它按处理的 GB 数量对查询进行分组,因此第一行是每个查询处理的 0 到 1 GB,第二行处理的是 1 到 2 GB,依此类推。
然后对于每一行,它显示处理的金额、计费金额以及相关成本和节省的成本。
这应该可以帮助您了解您的成本所在、您有多少个特定大小的查询以及您是否可以增加或降低您的预留大小。
请注意,BI 引擎只能加速某些 SELECT QUERY 而不能加速 MERGE、INSERT、CREATE 等语句。 还有更多的例外。 因此,为了公平比较,我排除了这些类型的查询,以便更好地了解节省的规模。 另请参阅: https://cloud.google.com/bigquery/docs/bi-engine-intro#bi-engine-use-cases
DECLARE QUERY_COST_PER_TB NUMERIC DEFAULT 5.00; -- current cost in dollars of processing 1 TB of data in BQ
with possible_bi_engine_jobs_incl_parent_jobs as (
select
creation_time,
bi_engine_statistics,
cache_hit,
total_bytes_processed / power(1024, 3) GB_processed,
floor(total_bytes_processed / power(1024, 3)) GB_processed_floor,
total_bytes_billed / power(1024, 3) GB_billed,
total_bytes_processed / power(1024, 4) * QUERY_COST_PER_TB expected_cost_in_euros,
total_bytes_billed / power(1024, 4) * QUERY_COST_PER_TB actual_cost_in_euros,
query,
job_id,
parent_job_id,
user_email,
job_type,
statement_type,
from `my_project_id.region-eu.INFORMATION_SCHEMA.JOBS`
where 1=1
and creation_time >= '2022-12-08'
and creation_time < '2022-12-09'
and cache_hit = false -- bi engine will not be improving on queries that are already cache hits
and total_bytes_processed is not null -- if there's no bytes processed, then ignore the job
and statement_type = 'SELECT' -- statement types such as MERGE, CREATE, UPDATE cannot be run by bi engine, only SELECT statements
and job_type = 'QUERY' -- LOAD jobs etc. cannot be run by bi engine, only QUERY jobs
and upper(query) like '%FROM%' -- query should contain FROM, otherwise it will not be run by bi engine
and upper(query) not like '%INFORMATION_SCHEMA.%' -- metadata queries can not be run by bi engine
),
-- to prevent double counting of total_bytes_processed and total_bytes_billed
parent_job_ids_to_ignore as (
select distinct parent_job_id
from possible_bi_engine_jobs_incl_parent_jobs
where parent_job_id is not null
),
possible_bi_engine_jobs_excl_parent_jobs as (
select *
from possible_bi_engine_jobs_incl_parent_jobs
where job_id not in (select parent_job_id from parent_job_ids_to_ignore) -- to prevent double counting of total_bytes_processed and total_bytes_billed
)
select
GB_processed_floor, -- all queries which processed less GB than the floor value
count(1) query_count,
sum(case when bi_engine_statistics.bi_engine_mode in ('FULL', 'PARTIAL') then 1 else 0 end) bi_engine_enabled,
sum(case when bi_engine_statistics.bi_engine_mode in ('DISABLED') or bi_engine_statistics.bi_engine_mode IS NULL then 1 else 0 end) bi_engine_disabled,
round(sum(GB_processed), 1) GB_processed,
round(sum(GB_billed), 1) GB_billed,
round(sum(expected_cost_in_euros), 2) expected_cost_in_euros,
round(sum(actual_cost_in_euros), 2) actual_cost_in_euros,
round(sum(expected_cost_in_euros) - sum(actual_cost_in_euros), 2) saved_cost
from possible_bi_engine_jobs_excl_parent_jobs
group by GB_processed_floor
order by GB_processed_floor
;
这将生成以下按查询大小分组的成本节省表:
有关 BI Engine 节省的其他有用链接:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.