在Eclipse中編譯Spark項目時出現問題

Question

我已經根據spark網站配置了所有內容，以啟動一個簡單的spark應用程序，該應用程序可以讀取和計數文件中的行並在另一個文件中顯示數字 。但是我無法運行該應用程序，因為我收到了很多東西錯誤 ，我不明白哪里出了問題。

這是我的項目結構：

sparkExamples
|-- pom.xml
`-- src
    |-- main/java/org/sparkExamplex/App.java
`-- resources
          |-- readLine
          |-- outputReadLine

的pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.sparkexamples</groupId>
    <artifactId>sparkExamples</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <name>sparkExamples</name>
    <url>http://maven.apache.org</url>


    <dependencies>
        <dependency> <!-- Spark dependency -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.3.1</version>
        </dependency>
    </dependencies>
</project>

App.java

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;

import scala.Tuple2;

import java.util.Arrays;
import java.util.regex.Pattern;

public final class App {
    private static final Pattern SPACE = Pattern.compile(" ");

    public static void main(String[] args) throws Exception {

        String inputFile = "resources/readLine";
        String outputFile = "resources/outputReadLine";
        // Create a Java Spark Context.
        SparkConf conf = new SparkConf().setAppName("wordCount").setMaster("spark://127.0.0.1:7077");
        JavaSparkContext sc = new JavaSparkContext(conf);
        // Load our input data.
        JavaRDD<String> input = sc.textFile(inputFile);
        // Split up into words.
        JavaRDD<String> words = input.flatMap(new FlatMapFunction<String, String>() {
            public Iterable<String> call(String x) {
                return Arrays.asList(x.split(" "));
            }
        });
        // Transform into word and count.
        JavaPairRDD<String, Integer> counts = words.mapToPair(new PairFunction<String, String, Integer>() {
            public Tuple2<String, Integer> call(String x) {
                return new Tuple2(x, 1);
            }
        }).reduceByKey(new Function2<Integer, Integer, Integer>() {
            public Integer call(Integer x, Integer y) {
                return x + y;
            }
        });
        // Save the word count back out to a text file, causing evaluation.
        counts.saveAsTextFile(outputFile);
    }
}


    }

顯示的錯誤：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/06/03 13:11:46 INFO SparkContext: Running Spark version 1.3.1
15/06/03 13:11:47 INFO SecurityManager: Changing view acls to: Administrator
15/06/03 13:11:47 INFO SecurityManager: Changing modify acls to: Administrator
15/06/03 13:11:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Administrator); users with modify permissions: Set(Administrator)
15/06/03 13:11:47 INFO Slf4jLogger: Slf4jLogger started
15/06/03 13:11:47 INFO Remoting: Starting remoting
15/06/03 13:11:47 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@mvettos.romelab.it.ibm.com:9164]
15/06/03 13:11:47 INFO Utils: Successfully started service 'sparkDriver' on port 9164.
15/06/03 13:11:47 INFO SparkEnv: Registering MapOutputTracker
15/06/03 13:11:47 INFO SparkEnv: Registering BlockManagerMaster
15/06/03 13:11:47 INFO DiskBlockManager: Created local directory at C:\Users\ADMINI~1\AppData\Local\Temp\spark-e6bb5cfc-6b96-4105-9a1c-843832ba60f9\blockmgr-dea7bb85-954c-4a4d-b3fb-74d7b6b1d9f5
15/06/03 13:11:47 INFO MemoryStore: MemoryStore started with capacity 467.6 MB
15/06/03 13:11:47 INFO HttpFileServer: HTTP File server directory is C:\Users\ADMINI~1\AppData\Local\Temp\spark-6e11c6bc-2743-4172-8d74-f3abc08d9f46\httpd-2bfa61a2-a1fd-4bd3-85c2-bcbc05d2ec27
15/06/03 13:11:47 INFO HttpServer: Starting HTTP Server
15/06/03 13:11:48 INFO Server: jetty-8.y.z-SNAPSHOT
15/06/03 13:11:48 INFO AbstractConnector: Started SocketConnector@0.0.0.0:9165
15/06/03 13:11:48 INFO Utils: Successfully started service 'HTTP file server' on port 9165.
15/06/03 13:11:48 INFO SparkEnv: Registering OutputCommitCoordinator
15/06/03 13:11:48 INFO Server: jetty-8.y.z-SNAPSHOT
15/06/03 13:11:48 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/06/03 13:11:48 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/06/03 13:11:48 INFO SparkUI: Started SparkUI at http://mvettos.romelab.it.ibm.com:4040
15/06/03 13:11:48 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@127.0.0.1:7077/user/Master...
15/06/03 13:11:49 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@127.0.0.1:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@127.0.0.1:7077
15/06/03 13:11:49 WARN Remoting: Tried to associate with unreachable remote 
address [akka.tcp://sparkMaster@127.0.0.1:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: no further information: /127.0.0.1:7077
15/06/03 13:12:08 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@127.0.0.1:7077/user/Master...
15/06/03 13:12:09 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@127.0.0.1:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@127.0.0.1:7077
15/06/03 13:12:09 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@127.0.0.1:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: no further information: /127.0.0.1:7077
15/06/03 13:12:28 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@127.0.0.1:7077/user/Master...
15/06/03 13:12:29 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@127.0.0.1:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@127.0.0.1:7077
15/06/03 13:12:29 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@127.0.0.1:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: no further information: /127.0.0.1:7077
15/06/03 13:12:48 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
15/06/03 13:12:48 WARN SparkDeploySchedulerBackend: Application ID is not initialized yet.
15/06/03 13:12:48 ERROR TaskSchedulerImpl: Exiting due to error from cluster scheduler: All masters are unresponsive! Giving up.

有人可以讓我怎么弄清楚這個嗎？

提前致謝。

Answer 1

經過幾次評論，我們可以將答案總結如下：

• 該組件已正確配置，您可以正確執行cloudera教程步驟。
• 該應用程序無法與Spark連接，因為您的系統中當前已安裝任何Spark服務。
• 因此，您現在需要一個教程來在Windows環境中獨立安裝Spark。 在stackoverflow搜索中，我發現了一個與此問題相同的問題 。

簡而言之，這個問題是重復的。 我希望這個答案可以對您有所幫助。