线程“main”中的异常java.io.IOException：无法运行程序“”：错误=2，没有这样的文件或目录

Question

I faced this issue when executing following script

./spark-submit /home/*****/public_html/****/****.py

I first used python3.7.2 and later python3.5.2 but still get the following error message.

Exception in thread "main" java.io.IOException: Cannot run program "": error=2, No such a file or directory.
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:100)
    at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: error=2, No such a file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)... 12 more`

Before it, I have several message output as

2019-02-07 11:30:18 WARN  Utils:66 - Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using xxx.xxx.xxx.xxx instead (on interface eth0)
2019-02-07 11:30:18 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2019-02-07 11:30:19 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I am able to exec python3 -VI am able to launch spark-shell and pyspark

And I feel very strange that no message is shown between "".

For my python code, it starts with

import sys
import urllib3
import requests

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import DoubleType, IntegerType, StringType

from CommonFunctions import *
from LanguageCodeParser import *

I also tried with a very simple python code

print("This is a test.")

Here are some message after execute bash -x spark-submit test.py

+ '[' -z /opt/spark-2.3.2-bin-hadoop2.7 ']'
+ export PYTHONHASHSEED=0
+ PYTHONHASHSEED=0
+ exec /opt/spark-2.3.2-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.SparkSubmit test.py

However, it doesn't work. Thanks for your helps in advance.

Answer 1

I found that setting PYSPARK_PYTHON=/usr/bin/python3 is useful

It will be good if this Environment Variable can be setup in

/opt/spark-2.3.2-bin-hadoop2.7/conf/spark-env.sh

Answer 2

as mentioned above, but in case of windows the spark-env.sh.template should be renamed to spark-env.cmd and the following should be inserted.

  set PYSPARK_PYTHON=C:\Users\usr\anaconda3\python.exe
  set PYSPARK_DRIVER_PYTHON=C:\Users\usr\anaconda3\python.exe
  set HADOOP_HOME=c:\winutils

please remove all comments starting with "#" in this file. Hope the above works for you as well