Loading data from S3 to PostgreSQL RDS

Question

We are planning to go for PostgreSQL RDS in AWS environment. There are some files in S3 which we will need to load every week. I don't see any option in AWS documentation where we can load data from S3 to PostgreSQL RDS. I see it is possible for Aurora but cannot find anything for PostgreSQL. Any help will be appreciated.

Answer 1

One option is to use AWS Data Pipeline. It's essentially a JSON script that allows you to orchestrate the flow of data between sources on AWS.

There's a template offered by AWS that's setup to move data between S3 and MySQL. You can find it here . You can easily follow this and swap out the MySQL parameters with those associated with your Postgres instance. Data Pipeline simply looks for RDS as the type and does not distinguish between MySQL and Postgres instances.

Scheduling is also supported by Data Pipeline, so you can automate your weekly file transfers.

To start this:

• Go to the Data Pipeline service in your AWS console
• Select "Build from template" under source
• Select the "Load S3 to MySQL table" template
• Fill in the rest of the fields and create the pipeline

From there, you can monitor the progress of the pipeline in the console!