Combining several datasets in a Netezza System

Question

We have a database with 65 tables that begin with Mon_Tues_Wed_201701 through Mon_Tues_Wed_202205. We create new tables with the naming convention of Mon_Tues_Wed_yyyymm so obviously it's growing. Each table contains about 1m-2m rows.

Here's what we want to do:

1. Union all Tables into one table
2. Do so automatically so all we have to do is hit the run statement -- no intervention or manual changes
3. Create a final table that is the union of the 65+ other views.
4. Bonus: if each table can have a column added (while being unioned) the name of the view (ie Source = 'Mon_Tues_Wed_202205' that would be appreciated.

We are currently doing this in SAS on a Grid System:

1. Importing the data from Netezza
2. Adding the Bonus column
3. Unioning the data
4. Exporting back to Netezza

Unfortunately this process takes 6 hours (1+ min per view import for a total time of 65+ min, 15 min to add the variables, 2+ hours to export to Netezza).

If you can show me code that would do the above, I'd really be grateful. I'm new to Netezza and it seems many of the rules from T-SQL processing do not apply.

Thank you

Paula

Answer 1

-- This can be done dynamically via a Stored Procedure
-- For example ...

    create or replace procedure my_dynamic_sql()
        returns integer
        language nzplsql
        execute as caller
    as begin_proc
    
    DECLARE
        tables RECORD;
        my_sql VARCHAR;
        union_statement VARCHAR;
        newline VARCHAR;
    BEGIN
    
        newline := chr(10);
        union_statement := '';
    
        my_sql := 'create table MON_TUES_WED_SUMMARY as' || newline ;
    
        raise notice 'Generating dynamic sql for';
    
        FOR tables IN
        SELECT TABLENAME FROM _V_TABLE WHERE OBJTYPE = 'TABLE' AND upper(TABLENAME) LIKE 'MON_TUES_WED_______' ORDER BY 1
        LOOP
    
            raise notice '     Table: %', tables.tablename;
    
            my_sql := my_sql || union_statement || 'select ''' || tables.tablename || '''::varchar(128) as source_table, * from ' || tables.tablename ;
    
            union_statement := ' union all' || newline ;
    
        END LOOP;
    
        raise notice 'Dropping the OLD results table';
        execute immediate 'DROP TABLE MON_TUES_WED_SUMMARY IF EXISTS;';
    
        raise notice 'The SQL looks like this %', my_sql;
    
        raise notice 'Executing the sql now ...';
        execute immediate my_sql;
    
        raise notice 'Done creating table MON_TUES_WED_SUMMARY';
    
    end;
    end_proc;
    
    --------------------------------------------------------------------------------
    
    -- Then, to invoke the stored procedure
    call my_dynamic_sql;