How to store data(tablenames) extracted from PG_TABLE_DEF into a table/temp table

Question

I will give you the context behind this request. This is related to this post Redshift cursor doesn't exist after creating a stored procedure . I have a workaround for this by using for loop with row_num window function. In order to do that I need to get the list of table names from PG_TABLE_DEF and store it in a temp table for processing through LOOP within Stored Proc. The challenge is we cannot run certain operations against table like PG_TABLE_DEF where it runs only on LEADER node. Hence I am getting this below error when i tried to copy data from PG_TABLE_DEF into a new temp table through CTAS .

ERROR: Specified types or functions (one per INFO message) not supported on Redshift tables.

Could someone please help to overcome this scenario.

Answer 1

As you state pg_table_def only exists on the leader node and on Redshift there is no way for a compute node to access this information during a query. So if you need this information on the compute nodes you need to first query it from the leader and then (somehow) route it back to the compute nodes. This can be done in several ways but all require that you fully execute the query on the leader node first.

You can do this with a Lambda function or other externally executed code that reads pg_table_def and then inserts (copys) the data into a normal table. OR you can execute the leader node query into a cursor and then read the cursor with a stored procedure depositing the data into a normal table. These 2 paths do basically the same thing, read the catalog table on the leader node and then put the result of this query into a normal table. I know of no other way to do this.

Here's an answer with code for doing this that I wrote up 2 years ago: How to join System tables or Information Schema tables with User defined tables in Redshift