__spark_conf__.zip: No such file or directory

Question

When I am submitting a Spark activity from Azure Data factory V2 on HDInsight cluster I am getting the error-

Application application_1524230454528_0060 failed 5 times due to AM Container for appattempt_1524230454528_0060_000005 exited with exitCode: -1000
    For more detailed output, check the application tracking page: http://hn1-rbc-cl.tsgjwtochfbexi5wvtaxuipmmg.rx.internal.cloudapp.net:8088/cluster/app/application_1524230454528_0060 Then click on links to logs of each attempt.
    Diagnostics: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    java.io.FileNotFoundException: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2732)
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2644)
    at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
    at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Failing this attempt. Failing the application.

Can someone help me figure out what the issue is or there is any other workaround to run spark jobs via Azure Data factory pipelines. I am stuck on this and could not proceed. As per Azure Data Factory documentation, I tried placing my files in ./files folder in the container where my spark jar is present. But still facing the same issue.

Answer 1

What's your HDI cluster version and the version of Java used to develop your program, please make sure the versions are the same.

BTW, HDI cluster 3.5+ require java 8, while the earlier versions require java 7.