How to submit spark job Remotely

Question

I have Nodejs server where spark is NOT installed and have spark with YARN setup in different server.

Requirement is to run spark job from Nodejs server remotely. Can someone help on this?.

Thanks for prompt response.

Answer 1

It's not possible to spawn process remotely. I suggest you the following ways,

1. Install spark where your Node server is running, and use this as client to point to your actual spark cluster. Your node server can use this client to trigger the job in client mode on the remote cluster.
2. You can setup a rest api on the spark cluster and let your node server hit an endpoint of this api which will trigger the job.

Elaborating the above answers, option 1 involves that spark is installed on both systems - the one with node server and the actual spark cluster. Spark on node server acts a client to the main spark cluster. Option 2 focuses on creating a rest API that handles triggers and these triggers initiate the spark job directly on the main cluster, this saves 2 installations.

Hope this helps.