aws将数据从s3加载到rds

Question

I have a question regarding technical architecture on AWS.

Situation: There are couple of sales units (each has Database in different location not connected with each other). Business requirement is that sales units place aggregated data in csv files which later will be loaded to report database.

I know already that I will need to do complex ETL processes (I work on SSIS), schedule jobs, write procedures and execute them automatically. Basically everything that MSSQL Server does + Data tools.

Question: Is it possible to load data securely to S3, then load to RDS (mssql) via ETL process exclusively on AWS? Is it a good idea? Can AWS GLUE/DataPipeline do the job?

If so please name the services with links how to do those tasks if possible.

Thank you for opinions.

Answer 1

Absolutely.

On a high level, within the data pipeline you would need -

1. S3 data node - Your input data
2. Activity - Any transformation you want to do
3. Resource - either EMR or EC2 depending on what resources/software is needed.
4. RDS data node - Output for the process, your RDS database.

In addition to the above, you can also setup retries, alerts for failures, success etc.

You can refer the AWS documents here - https://aws.amazon.com/documentation/data-pipeline/ https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/welcome.html