如何使用 AWS Glue 从 S3 存储桶合并 CSV 文件并将其保存回 S3

Question

Objective is to transform the data (csv files) from one S3 bucket to another S3 bucket - using Glue.

What I already tried:

I created a CSV classifier. I created a crawler which scans the data coming in S3 bucket. Where I am stuck:

Unable to find how can we store the output in S3 again without saving it in any RDS or other database services. Because Glue output is asking for database output, which I don't have and don't want to use.

Is there any way I can achieve the goal without using any other DB system, just plain - S3, Glue?

More Information Sample single CSV file, I am trying to merge

Classifier with delimeter of ";"

Crawler Configuration

Crawler Result (No schema detected)

Answer 1

The reason why Glue crawler detected schema is UNKNOWN because of the number of rows present in the source files. Refer to section Built-In CSV Classifier in this doc which you are using in your case.

According to the doc to be classified as CSV, the table schema must have at least two columns and two rows of data.

In your case you can use AWS Glue job and read files directly from S3 using either of below ways:

1.Create a dynamicframe and pass spearator as ; in format_options. Below is sample which you can modify according to your needs.

dyF = GlueContext.create_dynamic_frame_from_options(connection_type="s3",connection_options = {"paths": [InputDir]},format="csv",format_options={"withHeader": True,"separator": ";","quoteChar": '"',"escaper": '"'},transformation_ctx = "taxidata")

2.Use spark dataframe to read data from S3 and then convert it back to dynamicframe if you want to levarage Glue native transformations:

df = spark.read.options(delimiter=';').csv("s3://path-to-files/")

If you want to merge files with different schemas then read data containing different schema into different frames of your choice and then merge them using a Join operator.

Refer to this which has example code to join and write data back to s3.