使用SQL * Loader将数据发布到登台表中，然后发布到主数据库表中

Question

I am trying to load data into our main database table using SQL*Loader. Data will be provided into pipe separated CSV files.

I have developed a shell script to load data and it is working fine except one thing.

Here are the details and data to re-create the problem. Staging table structure in which data will be populated using SQL/*Loader:

create table stg_cmts_data (cmts_token varchar2(30), CMTS_IP varchar2(20));
create table stg_link_data (dhcp_token varchar2(30), cmts_to_add varchar2(200));
create table stg_dhcp_data (dhcp_token varchar2(30), DHCP_IP varchar2(20));

Data in CSV files; for stg_cmts_data in cmts_data.csv :

wnlb-cmts-01-1|10.15.0.1
wnlb-cmts-02-2|10.15.16.1
wnlb-cmts-03-3|10.15.48.1
wnlb-cmts-04-4|10.15.80.1
wnlb-cmts-05-5|10.15.96.1

For stg_dhcp_data in dhcp_map_03092015_1.csv :

dhcp-1-1-1|10.25.23.10,25.26.14.01
dhcp-1-1-2|56.25.111.25,100.25.2.01
dhcp-1-1-3|25.255.3.01,89.20.147.258
dhcp-1-1-4|10.25.26.36,200.32.58.69
dhcp-1-1-5|80.25.47.369,60.258.14.10

For stg_link_data in link_data.csv :

dhcp-1-1-1|wnlb-cmts-01-1,wnlb-cmts-02-2
dhcp-1-1-2|wnlb-cmts-03-3,wnlb-cmts-04-4,wnlb-cmts-05-5
dhcp-1-1-3|wnlb-cmts-01-1
dhcp-1-1-4|wnlb-cmts-05-8,wnlb-cmts-05-6,wnlb-cmts-05-0,wnlb-cmts-03-3
dhcp-1-1-5|wnlb-cmts-02-2,wnlb-cmts-04-4,wnlb-cmts-05-7
wnlb-dhcp-1-13|wnlb-cmts-02-2

Now after loading this data into the staging tables I have to populate the main database tables:

create table subntwk (subntwk_nm varchar2(20), subntwk_ip varchar2(30));
create table link (link_nm varchar2(50));

SQL scripts which I have created to load data:

spool load_cmts.log
set serveroutput on

DECLARE
  CURSOR c_stg_cmts IS SELECT *
                     FROM stg_cmts_data;
  TYPE t_stg_cmts IS TABLE OF stg_cmts_data%ROWTYPE INDEX BY pls_integer;
  l_stg_cmts t_stg_cmts;
  l_cmts_cnt       NUMBER;
  l_cnt            NUMBER;
  l_cnt_1          NUMBER;
 BEGIN
   OPEN c_stg_cmts;
   FETCH c_stg_cmts BULK COLLECT INTO l_stg_cmts;
   FOR i IN l_stg_cmts.FIRST..l_stg_cmts.LAST
   LOOP
      SELECT COUNT (1)
        INTO l_cmts_cnt
        FROM subntwk
       WHERE subntwk_nm = l_stg_cmts(i).cmts_token;

     IF l_cmts_cnt < 1 THEN
       INSERT
       INTO SUBNTWK
         (
           subntwk_nm
         )
         VALUES
        (
          l_stg_cmts(i).cmts_token      
         );

       DBMS_OUTPUT.PUT_LINE('Token has been added : '||l_stg_cmts(i).cmts_token);
     ELSE    
       DBMS_OUTPUT.PUT_LINE('Token is already present');
     END IF;

     EXIT WHEN l_stg_cmts.COUNT =0;
   END LOOP;

   commit;
 EXCEPTION
   WHEN OTHERS THEN
     DBMS_OUTPUT.put_line ('ERROR OCCURED ' || SQLERRM);
 END;
 /

 exit

For dhcp :

 spool load_dhcp.log
 set serveroutput on

 DECLARE
   CURSOR c_stg_dhcp IS SELECT *
                     FROM stg_dhcp_data;

   TYPE t_stg_dhcp IS TABLE OF stg_dhcp_data%ROWTYPE INDEX BY pls_integer;
   l_stg_dhcp t_stg_dhcp;
   l_dhcp_cnt       NUMBER;
   l_cnt            NUMBER;
   l_cnt_1          NUMBER;
 BEGIN
  OPEN c_stg_dhcp;
  FETCH c_stg_dhcp BULK COLLECT INTO l_stg_dhcp;
     FOR i IN l_stg_dhcp.FIRST..l_stg_dhcp.LAST
   LOOP

     SELECT COUNT (1)
      INTO l_dhcp_cnt
      FROM subntwk
     WHERE subntwk_nm = l_stg_dhcp(i).dhcp_token;
     IF l_dhcp_cnt < 1 THEN
       INSERT
      INTO SUBNTWK
        (
          subntwk_nm
        )
         VALUES
        (
          l_stg_dhcp(i).dhcp_token      
        );
       DBMS_OUTPUT.PUT_LINE('Token has been added : '||l_stg_dhcp(i).dhcp_token);
    ELSE    
      DBMS_OUTPUT.PUT_LINE('Token is already present');
     END IF;
    EXIT WHEN l_stg_dhcp.COUNT =0;
  END LOOP;

  commit;
 EXCEPTION
   WHEN OTHERS THEN
   DBMS_OUTPUT.put_line ('ERROR OCCURED ' || SQLERRM);
 END; 
/
exit

For link:

spool load_link.log
set serveroutput on

DECLARE
   l_cmts_1           VARCHAR2( 4000 CHAR );
   l_cmts_add         VARCHAR2( 200 CHAR );
   l_dhcp_cnt         NUMBER;
   l_cmts_cnt         NUMBER;
   l_link_cnt          NUMBER;
   l_add_link_nm      VARCHAR2( 200 CHAR );
 BEGIN
   FOR r IN (
        SELECT dhcp_token, cmts_to_add || ',' cmts_add
          FROM stg_link_data
       )
    LOOP

l_cmts_1      := r.cmts_add;
l_cmts_add    := TRIM(SUBSTR( l_cmts_1, 1, INSTR(l_cmts_1, ',' ) - 1 ));

SELECT COUNT(1)
  INTO l_dhcp_cnt
  FROM subntwk
 WHERE subntwk_nm = r.dhcp_token;

 IF l_dhcp_cnt = 0 THEN
  DBMS_OUTPUT.PUT_LINE('Device not found : '|| r.dhcp_token);
 ELSE
  WHILE l_cmts_add IS NOT NULL
  LOOP
    l_add_link_nm := r.dhcp_token||'_TO_'||l_cmts_add;
    SELECT COUNT(1)
      INTO l_cmts_cnt
      FROM subntwk
     WHERE subntwk_nm = TRIM(l_cmts_add);

    SELECT COUNT(1)
      INTO l_link_cnt
      FROM link
     WHERE link_nm = l_add_link_nm;

    IF l_cmts_cnt > 0 AND l_link_cnt = 0 THEN
      INSERT INTO link ( link_nm)
                VALUES ( l_add_link_nm);
      DBMS_OUTPUT.PUT_LINE( l_add_link_nm ||' '||'Has been added.');
    ELSIF l_link_cnt > 0 THEN
      DBMS_OUTPUT.PUT_LINE( 'link is already present : '|| l_add_link_nm );
   ELSIF l_cmts_cnt = 0 then
     DBMS_OUTPUT.PUT_LINE( 'NO CMTS FOUND for device to create link : '|| l_cmts_add );
    END IF;
  l_cmts_1   := TRIM(SUBSTR( l_cmts_1, INSTR( l_cmts_1, ',' ) + 1 ));
  l_cmts_add := TRIM(SUBSTR( l_cmts_1, 1, INSTR( l_cmts_1, ',' ) - 1 ));
END LOOP;
END IF;
  END LOOP;
  COMMIT;
EXCEPTION
   WHEN OTHERS THEN
    DBMS_OUTPUT.put_line ('ERROR OCCURED ' || SQLERRM);  
 END;
/
exit

Control files:

LOAD DATA
INFILE 'cmts_data.csv'
APPEND
INTO TABLE STG_CMTS_DATA
when (cmts_token != '') AND (cmts_token != 'NULL') AND (cmts_token != 'null')
  and (cmts_ip != '') AND (cmts_ip != 'NULL') AND (cmts_ip != 'null')
  FIELDS TERMINATED BY '|' OPTIONALLY ENCLOSED BY '"'
 TRAILING NULLCOLS
(cmts_token                "RTRIM(LTRIM(:cmts_token))",
 cmts_ip                   "RTRIM(LTRIM(:cmts_ip))")

For dhcp :

 LOAD DATA
 INFILE 'dhcp_data.csv'
 APPEND
 INTO TABLE STG_DHCP_DATA
 when (dhcp_token != '') AND (dhcp_token != 'NULL') AND (dhcp_token != 'null')
   and (dhcp_ip != '') AND (dhcp_ip != 'NULL') AND (dhcp_ip != 'null')
   FIELDS TERMINATED BY '|' OPTIONALLY ENCLOSED BY '"'
 TRAILING NULLCOLS
 (dhcp_token                "RTRIM(LTRIM(:dhcp_token))",
 dhcp_ip                   "RTRIM(LTRIM(:dhcp_ip))")

For link:

 LOAD DATA
 INFILE 'link_data.csv'
 APPEND
 INTO TABLE STG_LINK_DATA
 when (dhcp_token != '') AND (dhcp_token != 'NULL') AND (dhcp_token != 'null')
   and (cmts_to_add != '') AND (cmts_to_add != 'NULL') AND (cmts_to_add != 'null')
   FIELDS TERMINATED BY '|' OPTIONALLY ENCLOSED BY '"'
 TRAILING NULLCOLS
 (dhcp_token                   "RTRIM(LTRIM(:dhcp_token))",
  cmts_to_add          CHAR(4000) "RTRIM(LTRIM(:cmts_to_add))")

Shell script:

 if [ ! -d ./log ]
 then
   mkdir log
 fi

 if [ ! -d ./done ]
 then
    mkdir done
 fi

 if [ ! -d ./bad ]
 then
   mkdir bad
 fi

 nohup time sqlldr username/password@SID CONTROL=load_cmts_data.ctl    LOG=log/ldr_cmts_data.log BAD=log/ldr_cmts_data.bad DISCARD=log/ldr_cmts_data.reject ERRORS=100000 DIRECT=TRUE PARALLEL=TRUE &
 nohup time username/password@SID @load_cmts.sql

 nohup time sqlldr username/password@SID CONTROL=load_dhcp_data.ctl LOG=log/ldr_dhcp_data.log BAD=log/ldr_dhcp_data.bad DISCARD=log/ldr_dhcp_data.reject ERRORS=100000 DIRECT=TRUE PARALLEL=TRUE &                       
 nohup time sqlplus username/password@SID @load_dhcp.sql

 nohup time sqlldr username/password@SID CONTROL=load_link_data.ctl LOG=log/ldr_link_data.log BAD=log/ldr_link_data.bad DISCARD=log/ldr_link_data.reject ERRORS=100000 DIRECT=TRUE PARALLEL=TRUE &   
 nohup time sqlplus username/password@SID @load_link.sql

 mv *.log ./log

So here the problem which I encounter is while loading data into the link table I am checking if DHCP present in the subntwk table then proceed further else log error. If CMTS exits then create link else log error.

Now as we can we here multiple CMTS are associated with single DHCP.

So here in link table it is creating link but for the last iteration of the loop where I am fetching separate comma separated CMTS from stg_link_data table it is giving me log as CMTS not found.

For example

dhcp-1-1-1|wnlb-cmts-01-1,wnlb-cmts-02-2

Here I am supposed to create link for dhcp-1-1-1 with wnlb-cmts-01-1 and wnlb-cmts-02-2

Theses data are all present in the subntwk table but still it is giving me log as wnlb-cmts-02-2 NOT FOUND but we have already loaded it into subntwk table.

The same is happening with all CMTS from stg_link_data table which are in the last (I believe here you got what I am trying to explain).

But when I run the SQL scripts in the SQL Developer separately then it inserts all the valid links into the link table.

Here it should create 9 rows in the link table whereas now it is creating only 5 rows.

I use COMMIT in my script also but it is not helping me.

Please run these scripts in your machine and let me know if you are also getting the same behavior as I am getting, and please provide me some solution. I have tried lots of thing from yesterday but it is still same.

Here is the log for link table:

 link is already present : dhcp-1-1-1_TO_wnlb-cmts-01-1                         
 NO CMTS FOUND for device to create link : wnlb-cmts-02-2
 link is already present : dhcp-1-1-2_TO_wnlb-cmts-03-3                         
 link is already present : dhcp-1-1-2_TO_wnlb-cmts-04-4                         
 NO CMTS FOUND for device to create link : wnlb-cmts-05-5
 NO CMTS FOUND for device to create link : wnlb-cmts-01-1
 NO CMTS FOUND for device to create link : wnlb-cmts-05-8                       
 NO CMTS FOUND for device to create link : wnlb-cmts-05-6                       
 NO CMTS FOUND for device to create link : wnlb-cmts-05-0                       
 NO CMTS FOUND for device to create link : wnlb-cmts-03-3
 link is already present : dhcp-1-1-5_TO_wnlb-cmts-02-2                         
 link is already present : dhcp-1-1-5_TO_wnlb-cmts-04-4                         
 NO CMTS FOUND for device to create link : wnlb-cmts-05-7
 Device not found : wnlb-dhcp-1-13              

If more information is required then please let me know.

Answer 1

I figure out later in night that while loading into staging table using UNIX machine it was creating new line for each row. That is why the last CMTS was not found, For that i use DOS 2 UNIX conversion and it starts working perfectly.

That was dos2unix error !