AWS Glue 与 EMR 无服务器

Question

Recently, AWS announced Amazon EMR Serverless (Preview) https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/ - new very promising service.

From my understanding - AWS Glue is a managed service on top of Apache Spark (for transformation layer). AWS EMR is mostly used for Apache Spark as well. So EMR Serverless (for Apache Spark) looks like is something pretty much similar to AWS Glue.

Right now I have one question in my mind - what is the core difference from AWS Glue and when to choose EMR Serverless over Glue?

Potentially EMR Serverless, may be even a part of AWS Glue ecosystem for transformation layer? Maybe AWS is going to replace the transformation layer in AWS Glue with EMR Serverless, and then it may make sense. AWS Glue will play a role of ETL Overlay, Metastore with EMR Serverless as processing layer.

Answer 1

I'll give you my two cents about this because I've been wondering the same thing.

Glue

As per AWS documentation , AWS Glue is " Simple, scalable, and serverless data integration ". Glue can be used for a variety of things: as a metadata repository, automatic schema discovery, code generation, and run ETL pipelines to prepare data. Glue takes care of providing and managing the computation resources needed to run your data pipelines. Glue is a serverless service, so you don't need to create and manage the infrastructure, because Glue does it for you.

If we focus only on the processing feature and discard the Glue-specific features (schema discovery, code generation, etc) then EMR Serverless and Glue services look almost identical. One of the key advantages of both services is the ability to run Spark or Hive serverless applications.

What advantage will EMR Serverless have over Glue Spark jobs?

To run Glue, you must either specify MaxCapacity (for Glue version 1.0 or earlier jobs) or Worker type and the Number of workers (for Glue version 2.0 jobs). Both options assume, first, that there is some understanding of the data and workload per cluster, and second, that the workload during job execution will be uniform, ie, there will be no over- or under- utilization of the provisioned resources.

EMR Serverless

EMR Serverless is a new deployment option for AWS EMR. With EMR Serverless, you don't need to configure, optimize, protect, or manage clusters to run applications on these platforms. EMR Serverless helps you avoid over- or under-allocation of resources to process jobs at the individual stage level.

EMR Serverless automatically identifies the resources needed by jobs, provisions those resources to run the jobs, and releases them when the jobs are completed. In cases where applications require a response within seconds, such as interactive data analysis, the engineer can pre-initialize the necessary resources during application creation. This provides easy initialization, fast job startup, automatic capacity management, and simple cost control.

More info: https://luminousmen.com/post/emr-serverless-a-400level-guide

Answer 2

AWS Glue is a data integration service and ETL. Completely different service than EMR Analytics.

AWS Glue can be used as metadata store (table schema) for EMR and run integration jobs to prepare data (eg for the EMR). There are are data integration jobs and workflows. At least that's the intention to make the jobs limited, but simpler to manage.

EMR is much more (and very different). In theory the EMR could as well run the python data integration jobs in batch on top of a Spark cluster, but you could run any jobs inside a Spark cluster. EMR is more an analytics tool and processing tool. It is not limited to Spark processing of python batch jobs, you can use different frameworks. Though EMR serverless docs mention only Spark and Hive queries, you have much more control over the processing job.

If anything compares to the EMR service, it's Athena, which is something like EMR serverless with Spark and Presto and on its own network.