How to submit a Spark job on HDInsight via Powershell?

Question

Is there a way to submit a Spark job on HDInsight via Powershell? I know it can be done via activity in Azure Data Factory, but is there a way to submit python script to pyspark HDInsight from Powershell cmdlet?

Answer 1

Based on my knowledge, there is no Azure PowerShell command could do this.

You could use Apache Spark REST API, which is used to submit remote jobs to an Azure HDInsight Spark cluster. Please refer to this feedback .

HDInsight allows remote job submission through the REST API using Livy. It is part of the recent Spark release on Linux. https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-livy-rest-interface/