Spark EC2 support for Windows

Question

All the documentation about deploying a Spark cluster on Amazon EC2 is relative to Linux environments. However, my distributed project is, at this moment, dependent of some Windows functionalities, and I would like to start working with a Windows cluster while making the necessary changes.

I would like to know if there is any method that makes us able to deploy a Windows Spark cluster on EC2 in a way relatively similar to the spark-ec2 script provided by Spark .

Answer 1

spark-ec2 currently only supports launching clusters in EC2 using specific Linux AMIs , so deploying a Windows Spark cluster is currently not possible using that tool. I doubt that spark-ec2 will ever have that capability, since all of the setup scripts it uses assume a Linux host.

That said, Databricks recently announced a community-managed index of Spark packages , and people are adding in stuff there all the time. For example, there is already a package to let you launch Spark clusters on Google's Compute Engine .

Though there doesn't currently appear to be anything for you, I would keep my eye on that community index for something that lets you launch Windows Spark clusters on EC2.

Answer 2

In a resource with Spark Packages, suggested by Nick, you can see recently added project by Sigmoid Analytics - that lets you launch the Spark cluster on Azure - spark_azure :

https://github.com/sigmoidanalytics/spark_azure