堆栈内存溢出
  繁体   English   中英

Spark Maven项目未运行

[英]Spark Maven Project not running

 原文  2019-06-20 17:42:09  4   1    eclipse/ scala/ maven/ apache-spark

问题描述

我最近下载了Scala IDE for Eclipse for Spark / Scala项目。 为此，我创建了一个Maven项目并添加了Spark-core依赖项。 然后，我编写了基本的Spark代码，以创建数组之外的RDD并最终将其收集。 运行代码后，它在控制台上给我错误“ java.lang.NoClassDefFoundError：com / fasterxml / jackson / module / paranamer / ParanamerAnnotationIntrospector”。

火花代码： 

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

object Test {
  def main(args : Array[String]) : Unit = {
    println("Begining of the code")

    val sparkConf = new SparkConf()
    sparkConf.setAppName("New app")
    sparkConf.setMaster("local")

    val sc = new SparkContext(sparkConf)

    val array = Array(1,2,3,4,5)

    val arrayRDD = sc.parallelize(array, 2)

    arrayRDD.collect

    println("Ending of the code")
  }
}

Maven依赖关系： 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.talentorigin</groupId>
  <artifactId>SparkCourse</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <dependencies>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>2.4.3</version>
    </dependency>

  </dependencies>
</project>

控制台错误： 

Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/paranamer/ParanamerAnnotationIntrospector
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
    at org.apache.spark.SparkContext.parallelize(SparkContext.scala:716)
    at Test$.main(Test.scala:17)
    at Test.main(Test.scala)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.module.paranamer.ParanamerAnnotationIntrospector
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 6 more

Eclipse的问题标签： 

Failure to transfer com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9 from https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

Failure to transfer net.sf.py4j:py4j:jar:0.10.7 from https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced.

Missing artifact com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9

Missing artifact net.sf.py4j:py4j:jar:0.10.7

The container 'Maven Dependencies' references non existing library 'C:\Users\AniRudh\.m2\repository\com\fasterxml\jackson\module\jackson-module-paranamer\2.7.9\jackson-module-paranamer-2.7.9.jar'

我是与Maven合作的初学者。 请帮助我解决此问题。

1 个解决方案

解决方案1
 0  2019-06-20 19:16:47

我正在共享完整的pom文件。 您可以相应地更新pom文件。

包括以下内容以使构建正常工作：

  • 依赖火花，其他scala
  • 外挂程式

示例spark-scala-maven项目模板Github链接

这是完整的pom.xml 

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.techknowera</groupId>
    <artifactId>spark-batch</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>${project.artifactId}</name>
    <description>Spark Batch And Streaming Application</description>
    <inceptionYear>2019</inceptionYear>

    <properties>
        <scala.version>2.11</scala.version>
        <scala.full.version>2.11.8</scala.full.version>
        <spark.version>2.2.0</spark.version>
        <java.version>1.8</java.version>
        <jackson.version>2.6.5</jackson.version>
        <scala.maven.plugin.version>3.2.2</scala.maven.plugin.version>
        <maven.surefire.plugin.version>2.13</maven.surefire.plugin.version>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <!--Scala dependencies-->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.full.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.9</version>
            <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.scalatest/scalatest -->
        <dependency>
            <groupId>org.scalatest</groupId>
            <artifactId>scalatest_${scala.version}</artifactId>
            <version>3.0.5</version>
        </dependency>
        <!--Apache Spark Dependencies-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-yarn -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-yarn_${scala.version}</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.databricks</groupId>
            <artifactId>spark-xml_${scala.version}</artifactId>
            <version>0.4.1</version>
        </dependency>
    </dependencies>
    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-deploy-plugin</artifactId>
                <version>2.8.2</version>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
            <!-- disable surefire -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.7</version>
                <configuration>
                    <skipTests>true</skipTests>
                </configuration>
            </plugin>
            <!-- enable scala test -->
            <plugin>
                <groupId>org.scalatest</groupId>
                <artifactId>scalatest-maven-plugin</artifactId>
                <version>2.0.0</version>
                <configuration>
                    <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
                    <junitxml>.</junitxml>
                    <filereports>TestSuite.txt</filereports>
                </configuration>
                <executions>
                    <execution>
                        <id>test</id>
                        <goals>
                            <goal>test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <id>jar-with-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 在Eclipse上运行Maven项目 运行Maven项目 运行具有许多依赖项的Maven项目 运行Maven项目时出现问题 在glassfish服务器上运行Maven项目 在maven项目中运行scala应用程序 在Eclipse Luna中运行Maven项目 运行时在Maven项目中出现问题 带有Scala插件，Maven和Spark的Eclipse项目 如何在Eclipse中向Maven项目添加Spark？
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM