如何配置连接到AWS EMR spark集群的Java客户端

Question

I'm trying to write a simple spark application, and when i run it locally it works with setting the master as

.master("local[2]")

But after configuring spark cluster on AWS (EMR) i can't connet to the master url:

.master("spark://<master url>:7077")

Is this the way to do it? am i missing something here? The cluster is up and running, and when i tried adding my application as a step jar, so it will run directly in the cluster it worked. But i want to be able to run it from a remote machine.

would appreciate some help here, Thanks

Answer 1

To run from a remote machine, you will need to open the appropriate ports in the Security Group assigned to your EMR master node. You will need to add at least 7077.

If by "remote" you mean one that isn't in your AWS environment, you will also need to setup a way to route traffic to it from the outside.