Apache Spark AWS Glue 作业与 Hadoop 集群上的 Spark 在存储桶之间传输数据

Question

Let's say I need to transfer data between two S3 buckets in a manner of ETL and perform an easy transformation on the data during the transportation process (taking only part of the columns and filtering by ID). The data is parquet files and its size change between 1GB to 100GB.

What should be more efficient in terms of speed and cost - using an Apache Spark Glue job, or Spark on the Hadoop cluster with X machines?

Answer 1

The answer to this is basically the same for any serverless (Glue)/non-serverless (EMR) service equivalents.

The first should be faster to set up, but will be less configurable and probably more expensive. The second will give you more options for optimization (performance and cost) but you should not forget to include the cost of managing the service yourself. You can use AWS pricing calculator if you need some price estimate upfront.

I would definitely start with Glue and move to something more complicated if problems arise. Also, don't forget that there is serverless EMR now also available.

Answer 2

I read this question when determining if it was worthwhile to switch from AWS Glue to AWS EMR.

With configurable EC2 SPOT instances on EMR we drastically reduced a previous Glue job that read 1GB-4TB of csv uncompressed csv data. We were able to use spots instances to leverage much larger and faster Graviton processor EC2s that could load more data into RAM reducing spills to disk. Another benefit was that got rid of the dynamic frames which is very beneficial when you do not know a schema, but was overhead that we did not need. In addition the spot instances which are larger than what is provided by AWS Glue reduced our time to run but not too much. More importantly we cut our costs by 40-75%, yes that is even with the EC2 + EBS + EMR overhead cost per EC2 instance. We went from $25-250 dollars a day on Glue to $2-$60 on EMR. Costs monthly for this process was $1600 in AWS Glue and now is <$500. We run EMR as job_flow_run and TERMINATE when idle so that it essentially acts like Glue serverless.

We did not go with EMR Serverless because there was no spot instances which was probably the biggest benefit.

The only problem is that we did not switch earlier. We are now moving all AWS Glue jobs to AWS EMR.