how to add script actions to Azure HD Insight Cluster

Question

I am using Powershell to create an HD Insight Spark cluster and I would like to configure Spark and Hive configuration after cluster has been created using the Script Actions. The configuration values I would like to set are

spark.yarn.maxAppAttempts=1 spark.yarn.executor.memoryOverhead = 1152 hive.merge.orcfile.stripe.level = false Please note that I do not want to use the Ambari User Interface to set these values I would like to use Script actions.

Answer 1

See https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-customize-cluster-linux .

You can also use bootstrap. See https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-customize-cluster-bootstrap

Answer 2

In order to update hive.merge.orcfile.stripe.level u can use a bootstrap script like https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-customize-cluster-bootstrap . If u want to update spark params u have at least 3 options. First those params can be passed when u submit the spark app. Second u can update it using the Ambari UI as stated in : https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-resource-manager And Third u can update spark-defaults.conf using a script action or just by ssh into the head node and updating this file.