I am very new to AWS, and Postgresql.
I tried to use a data pipeline, but nothing seems to be available (template) for Postgres. I can't figure out how to connect to my RDS instance and import/export data from postgres.
I assumed that I could use EC2 to grab from my S3 bucket and import into Postgres in lieu of no data pipeline template being available. If it is possible I have no idea how.. Please advise if possible..
S3 -> RDS direct load is now possible for PostgreSQL Aurora and RDS PostgreSQL >= 11.1 as aws_s3
extension.
Parameters are similar to those of PostgreSQL COPY command
psql=> SELECT aws_s3.table_import_from_s3(
'table_name', '', '(format csv)',
'BUCKET_NAME', 'path/to/object', 'us-east-2'
);
Be warned that this feature does not work for older versions.
I wish AWS extends COPY command in RDS Postgresql as they did in Redshift. But for now they haven't and we have to do it by ourselves.
awscli
on your EC2 box (it might have been installed by default)awscli
with credentialsaws s3 sync
or aws s3 cp
commmands to download from s3 to your local directory\\COPY
the files into your RDS (requires \\
to copy from client directory)Example:
aws s3 cp s3://bucket/file.csv /mydirectory/file.csv
psql -h your_rds.amazonaws.com -U username -d dbname -c '\COPY table FROM ''file.csv'' CSV HEADER'
The prior answers have been superseded by more recent events at AWS.
There is now excellent support for S3-to-RDS-database loading via the Data Pipeline service (which can be used for many other data conversion tasks too, this is just one example).
This AWS article is for S3-to-RDS-MySQL. Should be very similar for RDS-Postgres.
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-template-copys3tords.html
if you can launch the psql client and connect to RDS on EC2 instance, you should be able to use the following command:
\\copy customer_orders from 'myfile.csv' with DELIMITER ','
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.