使用 AWS Glue 中的动态框架更新 RDS 表

Question

I have a glue job in that I am inserting data from csv to postgresql table. Now I need to update one row in the postgresql DB table. I have done this but the desired row is coming as a new entry in the table, not updating the existing one. How can I do this? Please help...

Answer 1

Glue currently does not support overwrite mode. You would need to convert your DynamicFrame to a DataFrame and then write with mode = overwrite like this:

dynamic_frame.toDF()
  .write
  .mode("overwrite")

Answer 2

Spark does not support updating of records yet. You can only overwrite(delete the existing records and add new) or append(add new records along with existing ones). However if you want to update a particular row you can use python library pg8000. Steps are as below for glue version 2.0 and python version 3:

1. Download and extract pg8000,asn1crypto and scramp tar files and then zip all of them into one.

2. Upload the file to s3 bucket

3. In python library path, add the path of the zipped pg8000 file. Eg: s3://bucketname/foldername/pg8000-1.19.2.zip

4. import only pg8000.native and use below code to make connection to the database directly.

   import ssl

   import pg8000.native

   conn = pg8000.native.Connection(database="database", host="xxxxxxrds.amazonaws.com", port=xxxx, user="user", password="password",ssl_context=ssl._create_unverified_context())

   insert_query = "your update query that you would generally write in postgresql"

   conn.run(insert_query)

   conn.run("COMMIT")

   conn.close()

PS: You can also use python '.format' to make your query generic.