在 Glue ETL 中执行存储过程

Question

How can we execute a SQL statement (like... 'call store_proc();') in Redshift via PySpark Glue ETL job by utilizing a catalog connection? I want to pass on the Redshift connection details (host, user, password) from Glue Catalog Connection.

I understand the 'write_dynamic_frame' option but I am not sure how to only execute a SQL statement against the Redshift server.

glueContext.write_dynamic_frame.from_jdbc_conf (frame=data_frame, catalog_connection="Redshift_Catalog_Conn", connection_options = {"preactions":"call stored_prod();","dbtable":"public.table1","database": "admin"}, redshift_tmp_dir="s3://glue_etl/")

Answer 1

As I understand, you want to call a Stored Procedure in RedShift from your Glue ETL Job. One way to do this is as follows: A simpler way to execute a stored procedure in Redshift is as follows.

post_query="begin; CALL sp_procedure1(); end;" 
datasink = glueContext.write_dynamic_frame.from_jdbc_conf(frame = mydf, \
                                    catalog_connection = "redshift_connection", \
                                    connection_options = {"dbtable": "my_table", "database": "dev","postactions":post_query}, \
                                    redshift_tmp_dir = 's3://tempb/temp/' transformation_ctx = "datasink")

The other more elaborate solution will be run SQL queries in application code.

1. Establish connection to your RedShift Cluster via Glue connections. Create dynamic frame in Glue with JDBC option.
   

 my_conn_options = { "url": "jdbc:redshift://host:port/redshift-database-name", "dbtable": "redshift-table-name", "user": "username", "password": "password", "redshiftTmpDir": args["TempDir"], "aws_iam_role": "arn:aws:iam::account id:role/role-name" } df = glueContext.create_dynamic_frame_from_options("redshift", my_conn_options)

2. Inorder to execute the stored procedure, we will use Spark SQL. So first convert Glue Dynamic Frame to Spark DF.

 spark_df=df.toDF() spark_df.createOrReplaceTempView("CUSTOM_TABLE_NAME") spark.sql('call store_proc();')

Your stored procedure in RedShift should have return values which can be written out to variables.