为什么我的语句由于资源繁忙异常而失败？

Question

As a part of an ETL process,I have this exchange subpartition statement:

ALTER TABLE DWH.QV_FACT_AMS EXCHANGE SUBPARTITION P08_2018_300_SALES WITH TABLE DWH.STG_QV_FACT_AMS;

(each run with different subpartition but the same 2 tables).

We started to get an exception:

ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired.

Which mean that some process update the table and we can't alter.

Since the final table - QV_FACT_AMS is only for reports, and the STG_QV_FACT_AMS is an inner-dynamic table that created in the ETL itself, I have no idea who can update those tables.

The ETL can't collide with another run of the ETL itself, so it can't be a trying to modify the same table from those 2 processes.

The exchange statement run a lot of times per day and finished ok, and the exception raise only around 9PM.

So I applied an AUDIT ALL on those 2 tables:

AUDIT ALL on dwh.qv_Fact_ams;
AUDIT ALL on dwh.stg_qv_Fact_ams;

But all the accesses to the tables didn't were on those hours and it was only select which doesn't supposed to lock the table.

example of the log of failure:

LOG_ID: 5879089
START_TIME: 18-07-2019 21:29:01
END_TIME: 18-07-2019 21:29:28
STATUS: FAILED
ORA_EXCEPTION: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

we can see that the exception was in 18-07-2019 21:29:28.

and this is the audit:

SELECT  * 
FROM DBA_MAINT.AUD$ 
WHERE OBJ$NAME IN ('QV_FACT_AMS','STG_QV_FACT_AMS') 
ORDER BY TIMESTAMP# DESC;

The only access except the etl itself is only action# = 3 (select), and an 3 hours later...

Answer 1

Set DDL_LOCK_TIMEOUT at the session level to find out who is blocking the command.

With the default settings, DDL on a locked table will immediately fail:

--Session #1: Insert but do not commit:
create table table1(a number);
insert into table1 values(1);

--Session #2:
--ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired
alter table table1 move;

Instead, we can tell Oracle to wait for a certain number of seconds instead of failing immediately.

--Session #2:
--The second command will hang for 9999 seconds, or until the lock is free.
alter session set ddl_lock_timeout = 9999;
alter table table1 move;

That gives us enough time to investigate what session and statement are blocking our commands. There are many ways to find blockers, and there may be many false positives. This code is a good starting point:

--Find potential blockers:
select sid, final_blocking_session, gv$session.*
from gv$session
where final_blocking_session is not null;

Once we have the FINAL_BLOCKING_SESSION (which is the SID for the blocking session), we can find out more information about that session, like what statement it's running. This SQL statement can help investigate the blocker:

--Use the FINAL_BLOCKING_SESSION from above.
select *
from gv$sql
join gv$session
    on gv$sql.sql_id = gv$session.sql_id
where sid = <final_blocking_session>;

Hopefully some of that information will make sense and reveal the culprit. Since the change didn't show up in the audit trail, I'm guessing there's a SYS user running a command, since SYS is not audited in the same place.