[英]Apache Spark Submit Error
I am new to Apache Spark and testing my first program. 我是Apache Spark的新手,正在测试我的第一个程序。
It is a 2-3 lines program just for testing purposes. 这是一个2-3行的程序,仅用于测试目的。
I am using Eclipse and compiled the java file with Maven. 我正在使用Eclipse,并使用Maven编译了Java文件。
I am trying to run the spark-submit but getting this error. 我正在尝试运行spark-submit,但出现此错误。
I do not think it is from the file name or the path. 我认为它不是来自文件名或路径。
Could it be from another issue? 可能是另一个问题吗?
...spark-2.1.0-bin-hadoop2.7\\bin>spark-submit --class "Main" --master local[4] "C:\\Users\\...\\target\\SparkTest-0.0.1-SNAPSHOT.jar"
The filename, directory name, or volume label syntax is incorrect. 文件名,目录名称或卷标签语法不正确。
This is the main class 这是主班
import java.util.Arrays;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.PairFunction;
public class SparkMain {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setMaster("local").setAppName("My App");
JavaSparkContext sc = new JavaSparkContext(conf);
System.out.println("HELLO");
JavaRDD<String> lines = sc.textFile("C:/spark/spark-2.1.0-bin-hadoop2.7/README.md");
System.out.println(lines.count());
}
}
You are passing a String to a Java program and on a Windows machine. 您正在将String传递给Java程序和Windows机器上。
Windows uses backslashes and need to be escaped. Windows使用反斜杠,需要转义。
I'm on a Mac, so this is hard to test, but you could try something like this. 我在Mac上,因此很难测试,但是您可以尝试类似的方法。
import java.nio.file.Paths;
...
String fileName = Paths.get("C:", "spark", "spark-2.1.0-bin-hadoop2.7", "README.md").toString()
JavaRDD<String> rdd = sc.textFile(fileName);
System.out.println(rdd.count());
If you want to be cross-platform, then perhaps this 如果您想跨平台,那么也许
String rootDir = Paths.get(System.getProperty("user.home")).getRoot().toString();
String fileName = Paths.get(rootDir, "spark", ...);
...
Refer: Java Essentials | 参考: Java Essentials | Path Operations
路径操作
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.