AWS Glue - 从 sql server 表中读取并作为自定义 CSV 文件写入 S3

Question

I am working on Glue since january, and have worked multiple POC, production data lakes using AWS Glue / Databricks / EMR, etc. I have used AWS Glue to read data from S3 and perform ETL before loading to Redshift, Aurora, etc.

I have a need now to read data from a source table which is on SQL SERVER, and fetch data, write to a S3 bucket in a custom (user defined) CSV file, say employee.csv.

Am looking for some pointers, to do this please.

Thanks

Answer 1

You can connect using JDBC specifying connectionType=sqlserver to get a dynamic frame connecting to SQL SERVER. See here for GlueContext docs

dynF = glueContext.getSource(connection_type="sqlserver", url = ..., dbtable=..., user=..., password=)

Answer 2

This task fits AWS DMS (Data Migration Service) use case. DMS is designed to either migrate data from one data storage to another or keep them in sync. It can certainly keep in sync as well as transform your source (ie, MSSQL) to your target (ie, S3).

There is one non-negligible constraint in your case thought. Ongoing sync with MSSQL source only works if your license is the Enterprise or Developer Edition and for versions 2016-2019.