[英]Reduce the execution time of large query
My query takes 30+ minutes to process this query.我的查询需要 30 多分钟来处理此查询。 It indeed works on very large data set, however I may be missing something basic which can reduce the execution time.它确实适用于非常大的数据集,但是我可能会遗漏一些可以减少执行时间的基本知识。
Query works on many reducer stages each using 1000+ reducers. Query 在许多 reducer 阶段工作,每个阶段使用 1000 多个 reducer。 Runs on Tez engine.在 Tez 引擎上运行。
I tried to enable CBO but no luck, also tried to limit reducers to 500 but execution time remains high.我尝试启用 CBO 但没有运气,也尝试将减速器限制为 500,但执行时间仍然很长。
select itt.tr_date, sum (bkt_sum_pc) as pts
from itops_trxn itt,
( select acttrxnID, max(act_cmp_id) as act_cmp_id
from itops_trxn_act a, ll_act_act_trxn b where a.act_trxn_ID = b.ACOUNTtrxnID group by acttrxnID
) A,
(select cmp_id, max (cmp_name) as name
from itops_offer group by cmp_id
) c
where itt.acttrxnID = A.acttrxnID and act_cmp_id = c.cmp_id
and itt.type = 'ajstmnt'
and itt.event_header_event_name NOT IN ('composite.sys.act.merge', 'pos.sys.identity', 'composite.sys.act.pcmerge')
and itt.event_atomic_operation_type = 'CT'
and itt.tr_date >='2018-10-31'
group by itt.tr_date, channel, location_storeparentid, meta_trxnreason, act_cmp_id,name;
Rewrite joins explicitly and move these conditions显式重写连接并移动这些条件
where itt.acttrxnID = A.acttrxnID and act_cmp_id = c.cmp_id
to the join ON clause:加入 ON 子句:
select itt.tr_date, sum (bkt_sum_pc) as pts
from itops_trxn itt
INNER JOIN
( select acttrxnID, max(act_cmp_id) as act_cmp_id
from itops_trxn_act a, ll_act_act_trxn b where a.act_trxn_ID = b.ACOUNTtrxnID group by acttrxnID
) A ON itt.acttrxnID = A.acttrxnID
INNER JOIN
(select cmp_id, max (cmp_name) as name
from itops_offer group by cmp_id
) c ON A.act_cmp_id = c.cmp_id
where itt.type = 'ajstmnt'
and itt.event_header_event_name NOT IN ('composite.sys.act.merge', 'pos.sys.identity', 'composite.sys.act.pcmerge')
and itt.event_atomic_operation_type = 'CT'
and itt.tr_date >='2018-10-31'
group by itt.tr_date, channel, location_storeparentid, meta_trxnreason, act_cmp_id,name;
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.