Oracle DB: Portioning delete with Automation
Question
I'm new to work with database. Describing the Problem below: Problem: Having 'X' number of Tables with 'Y' number of partitions. Now want to have a way of finding which tables have partitions and which partitions are almost full/partitions created before D-30 is needed to be deleted as getting error in application due to such issue.
Please share guidance/queries how this problem can be resolved?
Artifacts : Tried several queries as SYSDBA to list down all the partitions but there is multiple dependencies pop up. Still trying to figuring a concrete way to make a scheduler Job.
1 answers
solution1
0 2022-08-08 18:37:49
step 1 is to create your PARTITIONS. In my test CASE I am creating 1 PARTITION t1= weekly but I'm leaving the syntax for other partitions:
t0 = daily, t2= monthly, t3=quaterly and t4=yearly.
Then I will populate PARTITION t1 with sample data. I'm using 1 date for each day but during your testing add as many dates as you like per day. I would suggest data to emulate your application.
In addition, I am using a GLOBAL index for table t1 and the rest are LOCAL indexes.
When a partition from a global index is dropped the entire index MUST be rebuilt or it will be invalid.
Based on your application you need to make your own determination whether to use global or local indexes!!
Note: I am creating all the PARTITIONS on DATE columns but you can also use TIMESTAMP columns If need be.
/* weekly PARTITION */
CREATE TABLE t1 (
seq_num NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 1) NOT NULL,
dt DATE)
PARTITION BY RANGE (dt)
INTERVAL ( NUMTODSINTERVAL (7, 'DAY') ) (
PARTITION OLD_DATA VALUES LESS THAN (DATE '2022-04-01')
);
/
INSERT into t1 (dt)
with dt (dt, interv) as (
select date '2022-04-01', numtodsinterval(1,'DAY') from dual
union all
select dt.dt + interv, interv from dt
where dt.dt + interv < date '2022-08-31')
select dt from dt;
/
create index t1_global_ix on T1 (dt);
/
/* daily PARTITION */
CREATE TABLE t0 (
seq_num NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 1) NOT NULL,
dt DATE)
PARTITION BY RANGE (dt)
INTERVAL ( NUMTODSINTERVAL (1, 'DAY') ) (
PARTITION OLD_DATA VALUES LESS THAN (DATE 2022-01-01')
);
/
INSERT into t0 (dt)
with dt (dt, interv) as (
select date '2022-01-01', numtodsinterval(1,'DAY') from dual
union all
select dt.dt + interv, interv from dt
where dt.dt + interv < date '2022-12-31')
select dt from dt;
/
create index t0_global_ix on T0 (dt);
/
/* MONTHLY PARTITION */
CREATE TABLE t2 (
seq_num NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 1) NOT NULL,
dt DATE)
PARTITION BY RANGE (dt)
INTERVAL ( NUMTODSINTERVAL (1, 'MONTH') ) (
PARTITION OLD_DATA VALUES LESS THAN (DATE '2022-01-01')
);
/
INSERT into t2 (dt)
with dt (dt, interv) as (
select date '2021-01-01', numtodsinterval(1,'DAY') from dual
union all
select dt.dt + interv, interv from dt
where dt.dt + interv < date '2022-12-31')
select dt from dt;
/
create index t2_local_ix on T2 (dt) LOCAL;
/
/* QUARTERLY PARTITION */
CREATE TABLE t3 (
seq_num NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 1) NOT NULL,
dt DATE
)
PARTITION BY RANGE (dt)
INTERVAL
(NUMTOYMINTERVAL(3,'MONTH'))
(
PARTITION OLD_DATA values LESS THAN (TO_DATE('2021-01-01','YYYY-MM-DD'))
);
/
INSERT into t3 (dt)
with dt (dt, interv) as (
select date '2021-01-01', numtodsinterval(1,'DAY') from dual
union all
select dt.dt + interv, interv from dt
where dt.dt + interv < date '2022-12-31')
select dt from dt;
/
create index t3_local_ix on T3 (dt) LOCAL;
/
/* yearly PARTITION */
CREATE TABLE t4 (
seq_num NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 1) NOT NULL,
dt DATE
)
PARTITION BY RANGE (dt)
INTERVAL
(NUMTOYMINTERVAL(1,'YEAR'))
(
PARTITION OLD_DATA values LESS THAN (TO_DATE('2020-01-01','YYYY-MM-DD'))
);
/
INSERT into t4 (dt)
with dt (dt, interv) as (
select date '2021-01-01', numtodsinterval(90,'DAY') from dual
union all
select dt.dt + interv, interv from dt
where dt.dt + interv < date 2023-12-31')
select dt from dt;
/
create index t4_local_ix on T4 (dt) LOCAL;
/
Every time a new partition is created Oracle generates a system name, which is quite cryptic
Here is a list of the PARTITION names, which Oracle generated when I loaded the data above
SELECT PARTITION_NAME
FROM USER_TAB_PARTITIONS
WHERE TABLE_NAME = 'T1'
PARTITION_NAME
OLD_DATA
SYS_P458773
SYS_P458774
SYS_P458775
SYS_P458776
SYS_P458777
SYS_P458778
SYS_P458779
SYS_P458780
SYS_P458781
SYS_P458782
SYS_P458783
SYS_P458784
SYS_P458785
SYS_P458786
SYS_P458787
SYS_P458788
SYS_P458789
SYS_P458790
SYS_P458791
SYS_P458792
SYS_P458793
SYS_P458794
Although PARTITION management will work with system GENERATED PARTITION names, I use the procedure below to rename them to something more meaningful.
Let's create and run the procedure and take a look at the names again. As you can see, since we are working with weekly partitions the name is P_ for partittion YYYY 4 digit year the PARTITION is in, W for Week of the year, and ## for the week number within the year.
I would suggest using the scheduler to run this process at least once a day. You can run it as many time as you want as it will not cause any harm.
CREATE OR REPLACE PROCEDURE MaintainPartitions IS EXPRESSION_IS_OF_WRONG_TYPE EXCEPTION;
PRAGMA EXCEPTION_INIT(EXPRESSION_IS_OF_WRONG_TYPE, -6550);
CURSOR PartTables IS
SELECT TABLE_NAME, INTERVAL
FROM USER_PART_TABLES
WHERE PARTITIONING_TYPE = 'RANGE'
ORDER BY TABLE_NAME;
CURSOR TabParts(aTableName VARCHAR2) IS
SELECT PARTITION_NAME, HIGH_VALUE
FROM USER_TAB_PARTITIONS
WHERE regexp_like(partition_name,'^SYS_P[[:digit:]]{1,10}') AND
TABLE_NAME = aTableName AND
table_name not like 'BIN$%'
and interval is not null
ORDER BY PARTITION_POSITION;
ym INTERVAL YEAR TO MONTH;
ds INTERVAL DAY TO SECOND;
newPartName VARCHAR2(30);
PERIOD TIMESTAMP;
BEGIN
FOR aTab IN PartTables LOOP
BEGIN
EXECUTE IMMEDIATE 'BEGIN :ret := '||aTab.INTERVAL||'; END;' USING OUT ds;
ym := NULL;
EXCEPTION
WHEN EXPRESSION_IS_OF_WRONG_TYPE THEN
EXECUTE IMMEDIATE 'BEGIN :ret := '||aTab.INTERVAL||'; END;' USING OUT ym;
ds := NULL;
END;
FOR aPart IN TabParts(aTab.TABLE_NAME) LOOP
EXECUTE IMMEDIATE 'BEGIN :ret := '||aPart.HIGH_VALUE||'; END;' USING OUT PERIOD;
IF ds IS NOT NULL THEN
IF ds >= INTERVAL '7' DAY THEN
-- Weekly partition
EXECUTE IMMEDIATE 'BEGIN :ret := TO_CHAR('||aPart.HIGH_VALUE||' - :int, :fmt); END;' USING OUT newPartName, INTERVAL '1' DAY, '"P_"IYYY"W"IW';
ELSE
-- Daily partition
EXECUTE IMMEDIATE 'BEGIN :ret := TO_CHAR('||aPart.HIGH_VALUE||' - :int, :fmt); END;' USING OUT newPartName, INTERVAL '1' DAY, '"P_"YYYYMMDD';
END IF;
ELSE
IF ym = INTERVAL '3' MONTH THEN
-- Quarterly partition
EXECUTE IMMEDIATE 'BEGIN :ret := TO_CHAR('||aPart.HIGH_VALUE||' - :int, :fmt); END;' USING OUT newPartName, INTERVAL '1' DAY, '"P_"YYYY"Q"Q';
ELSE
-- Monthly partition
EXECUTE IMMEDIATE 'BEGIN :ret := TO_CHAR('||aPart.HIGH_VALUE||' - :int, :fmt); END;' USING OUT newPartName, INTERVAL '1' DAY, '"P_"YYYYMM';
END IF;
END IF;
IF newPartName <> aPart.PARTITION_NAME THEN
EXECUTE IMMEDIATE 'ALTER TABLE '||aTab.TABLE_NAME||' RENAME PARTITION '||aPart.PARTITION_NAME||' TO '||newPartName;
END IF;
END LOOP;
END LOOP;
END MaintainPartitions;
/
EXEC MaintainPartitions
SELECT PARTITION_NAME
FROM USER_TAB_PARTITIONS
WHERE TABLE_NAME = 'T1'
PARTITION_NAME
OLD_DATA
P_2022W14
P_2022W15
P_2022W16
P_2022W17
P_2022W18
P_2022W19
P_2022W20
P_2022W21
P_2022W22
P_2022W23
P_2022W24
P_2022W25
P_2022W26
P_2022W27
P_2022W28
P_2022W29
P_2022W30
P_2022W31
P_2022W32
P_2022W33
P_2022W34
SELECT COUNT(*) FROM USER_TAB_PARTITIONS
COUNT(*)
31
Next step is setting up your RETENTION table. There should be an entry for each interval range PARTITION.
The RETENTION value is for you to decide. In my example, I chose 30 days fir table T1. This means, when the high value for a PARTITION is greater than 30 days its eligible to be dropped. So chose wisely when setting up these values.
Note: I listed the names of other tables to show how each table has its own value.
CREATE TABLE PARTITION_RETENTION (
seq_num NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 1) NOT NULL,
TABLE_NAME VARCHAR2(30),
RETENTION INTERVAL DAY(3) TO SECOND(0),
CONSTRAINT
partition_retention_pk primary key (table_name),
CONSTRAINT CHK_NON_ZERO_DAYS CHECK (
RETENTION > INTERVAL '0' DAY
),
CONSTRAINT CHK_WHOLE_DAYS CHECK (
EXTRACT(HOUR FROM RETENTION) = 0
AND EXTRACT(MINUTE FROM RETENTION) = 0
AND EXTRACT(SECOND FROM RETENTION) = 0
)
);
insert into PARTITION_RETENTION (TABLE_NAME, RETENTION)
select 'T0', interval '10' day from dual union all
select 'T1', interval '30' day from dual union all
select 'T2', interval '15' day from dual union all
select 'T3', interval '30' day from dual union all
select 'T4', 15 * interval '1' day from dual union all
select 'T5', 5 * interval '1 00:00:00' day to second from dual;
Below are 3 procedures that need to be created.
The ddl procedure is a wrapper, which shows you what is being processed and how long it takes.
The rebuild_index procedure is obvious it rebuilds any invalid indexes. As I mentioned above if you are using a global index and a PARTITION is dropped then the index needs to be rebuilt. I hardcoded parallel 4 in this example but if you have plenty if CPU power you may want to increase the number to fit your needs.
In addition, there are other ways indexes can be marked unusable so you may want to consider scheduling that task.
Lastly, is the anonymous block. Which actually drops the PARTITIONS for, which the retention PERIOD has passed. This needs to be scheduled once a day!!
If you look carefully at the anonymous block the last step is a call to the rebuild index procedure. So if an index is unusable it will be rebuilt.
Now let's run the process and see what happens.
CREATE OR REPLACE PROCEDURE ddl(p_cmd varchar2)
authid current_user
is
t1 pls_integer;
BEGIN
t1 := dbms_utility.get_time;
dbms_output.put_line(p_cmd);
execute immediate p_cmd;
dbms_output.put_line((dbms_utility.get_time - t1)/100 || ' seconds');
END;
/
CREATE OR REPLACE PROCEDURE rebuild_index
authid current_user
is
BEGIN
for i in (
select index_owner, index_name, partition_name, 'partition' ddl_type
from all_ind_partitions
where status = 'UNUSABLE'
union all
select owner, index_name, null, null
from all_indexes
where status = 'UNUSABLE'
)
loop
if i.ddl_type is null then
ddl('alter index '||i.index_owner||'.'||i.index_name||' rebuild parallel 4 online');
else
ddl('alter index '||i.index_owner||'.'||i.index_name||' modify '||i.ddl_type||' '||i.partition_name||' rebuild parallel 4 online');
end if;
end loop;
END;
/
DECLARE
CANNOT_DROP_LAST_PARTITION EXCEPTION;
PRAGMA EXCEPTION_INIT(CANNOT_DROP_LAST_PARTITION, -14758);
CANNOT_DROP_ONLY_ONE_PARTITION EXCEPTION;
PRAGMA EXCEPTION_INIT(CANNOT_DROP_ONLY_ONE_PARTITION, -14083);
ts TIMESTAMP;
CURSOR TablePartitions IS
SELECT TABLE_NAME, PARTITION_NAME, p.HIGH_VALUE, t.INTERVAL, RETENTION, DATA_TYPE
FROM USER_PART_TABLES t
JOIN USER_TAB_PARTITIONS p USING (TABLE_NAME)
JOIN USER_PART_KEY_COLUMNS pk ON pk.NAME = TABLE_NAME
JOIN USER_TAB_COLS tc USING (TABLE_NAME, COLUMN_NAME)
JOIN PARTITION_RETENTION r USING (TABLE_NAME)
WHERE pk.object_type = 'TABLE' AND
t.partitioning_type = 'RANGE' AND
REGEXP_LIKE (tc.data_type, '^DATE$|^TIMESTAMP.*');
BEGIN
FOR aPart IN TablePartitions LOOP
EXECUTE IMMEDIATE 'BEGIN :ret := '||aPart.HIGH_VALUE||'; END;' USING OUT ts;
IF ts < SYSTIMESTAMP - aPart.RETENTION THEN
BEGIN
ddl('alter table '||aPart.TABLE_NAME||' drop partition '||aPart.partition_name);
EXCEPTION
WHEN CANNOT_DROP_ONLY_ONE_PARTITION THEN
DBMS_OUTPUT.PUT_LINE('Cant drop the only partition '||aPart.PARTITION_NAME ||' from table '||aPart.TABLE_NAME);
ddl('ALTER TABLE '||aPart.TABLE_NAME||' TRUNCATE PARTITION '||aPart.PARTITION_NAME);
WHEN CANNOT_DROP_LAST_PARTITION THEN
BEGIN
DBMS_OUTPUT.PUT_LINE('Drop last partition '||aPart.PARTITION_NAME ||' from table '||aPart.TABLE_NAME);
EXECUTE IMMEDIATE 'ALTER TABLE '||aPart.TABLE_NAME||' SET INTERVAL ()';
ddl('alter table '||aPart.TABLE_NAME||' drop partition '||aPart.partition_name);
EXECUTE IMMEDIATE 'ALTER TABLE '||aPart.TABLE_NAME||' SET INTERVAL( '||aPart.INTERVAL||' )';
EXCEPTION
WHEN CANNOT_DROP_ONLY_ONE_PARTITION THEN
-- Depending on the order the "last" partition can be also the "only" partition at the same time
EXECUTE IMMEDIATE 'ALTER TABLE '||aPart.TABLE_NAME||' SET INTERVAL( '||aPart.INTERVAL||' )';
DBMS_OUTPUT.PUT_LINE('Cant drop the only partition '||aPart.PARTITION_NAME ||' from table '||aPart.TABLE_NAME);
ddl('ALTER TABLE '||aPart.TABLE_NAME||' TRUNCATE PARTITION '||aPart.PARTITION_NAME);
END;
END;
END IF;
END LOOP;
rebuild_index();
END;
alter table T1 drop partition OLD_DATA
.02 seconds
alter table T1 drop partition P_2022W14
.01 seconds
alter table T1 drop partition P_2022W15
.02 seconds
alter table T1 drop partition P_2022W16
.01 seconds
alter table T1 drop partition P_2022W17
.02 seconds
alter table T1 drop partition P_2022W18
.01 seconds
alter table T1 drop partition P_2022W19
.02 seconds
alter table T1 drop partition P_2022W20
.01 seconds
alter table T1 drop partition P_2022W21
.01 seconds
alter table T1 drop partition P_2022W22
.02 seconds
alter table T1 drop partition P_2022W23
.01 seconds
alter table T1 drop partition P_2022W24
.01 seconds
alter table T1 drop partition P_2022W25
.01 seconds
alter table T1 drop partition P_2022W26
.01 seconds
alter table T1 drop partition P_2022W27
.02 seconds
alter index SQL_WUKYPRGVPTOUVLCAEKUDCRCQI.T1_GLOBAL_IX rebuild parallel 4 online
.1 seconds
…
…
…
alter index SQL_WUKYPRGVPTOUVLCAEKUDCRCQI.T1_GLOBAL_IX rebuild parallel 4 online
.1 seconds
SELECT count(*) from USER_TAB_PARTITIONS
Where
table_name not like 'BIN$%'
8
SELECT PARTITION_NAME
FROM USER_TAB_PARTITIONS
WHERE TABLE_NAME = 'T1'
AND
table_name not like 'BIN$%'
P_2022W28
P_2022W29
P_2022W30
P_2022W31
P_2022W32
P_2022W33
P_2022W34
P_2022W35
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.