堆栈内存溢出
  简体   繁体   English

Maven 构建错误（Scala + Spark）：object apache 不是 ZEFE90A8E604A7C8DBB7D88 的成员

[英]Maven build ERROR (Scala + Spark):object apache is not a member of package org

 原文  2020-05-28 09:28:50  9   1    scala/ maven/ apache-spark/ log4j

问题描述

I read several threads about this problem on StackOverflow and other resources ( 1st , 2nd , 3rd ), but unfortunately it doesn't help.我在 StackOverflow 和其他资源（ 1st 、 2nd 、 3rd ）上阅读了几个关于这个问题的帖子，但不幸的是它没有帮助。 Also almost all of them describes the same problem on SBT, not Maven.几乎所有人都在 SBT 上描述了相同的问题，而不是 Maven。

I also checked the Scala/Spark compatibility in Spark docs ( here ) and it seems like I have correct versions (Scala 2.11.8 + Spark 2.2.0)我还检查了 Spark 文档（此处）中的 Scala/Spark 兼容性，似乎我有正确的版本（Scala 2.11.8 + Spark 2.2.0）

I will describes my full workflow, because I'm not sure what information could be helpful to establish the root cause.我将描述我的完整工作流程，因为我不确定哪些信息可能有助于确定根本原因。

This is the code I'm trying to build这是我正在尝试构建的代码

pom.xml: pom.xml：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.example</groupId>
    <artifactId>SparkWordCount</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>SparkWordCount</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>

        <plugins>
            <!-- mixed scala/java compile -->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.3.1</version>
                <executions>
                    <execution>
                        <id>compile</id>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <phase>compile</phase>
                    </execution>
                    <execution>
                        <id>test-compile</id>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                        <phase>test-compile</phase>
                    </execution>
                    <execution>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <!-- for fatjar -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.1.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <finalName>uber-SparkWordCount-1.0-SNAPSHOT</finalName>
                    <appendAssemblyId>false</appendAssemblyId>
                </configuration>
                <executions>
                    <execution>
                        <id>assemble-all</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.2.0</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <mainClass>fully.qualified.MainClass</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.2.0</version>
            <scope>provided</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.2.0</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</project>

SparkWordCount.scala SparkWordCount.scala


import org.apache.spark.sql.SparkSession
import org.apache.log4j.Logger
import org.apache.log4j.Level

object SparkWordCount {
  def main(args: Array[String]) {

    Logger.getLogger("org").setLevel(Level.ERROR)

    val spark = SparkSession
      .builder()
      .appName("SparkSessionZipsExample")
      .master("local")
      .getOrCreate()

    val myRdd = spark.sparkContext.parallelize(List(1,2,3,4,5,6,7))
    myRdd.foreach(number => println("Lol, this is number = " + number))
  }
}

It works fine when I just launch the main() method:当我启动 main() 方法时它工作正常：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Lol, this is number = 1
Lol, this is number = 2
Lol, this is number = 3
Lol, this is number = 4
Lol, this is number = 5
Lol, this is number = 6
Lol, this is number = 7

Then I tried to use SparkSQL DataFrame:然后我尝试使用 SparkSQL DataFrame：

import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.log4j.Logger
import org.apache.log4j.Level

object SparkWordCount {
  def main(args: Array[String]) {

    Logger.getLogger("org").setLevel(Level.ERROR)

    val spark = SparkSession
      .builder()
      .appName("SparkSessionZipsExample")
      .master("local")
      .getOrCreate()

    val airports: DataFrame = spark.read.csv("C:\\Users\\Евгений\\Desktop\\DATALEARN\\airports.csv")
    airports.show()
  }
}

And this code throws an error:这段代码会引发错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Logger
    at SparkWordCount$.main(SparkWordCount.scala:10)
    at SparkWordCount.main(SparkWordCount.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 2 more

Process finished with exit code 1

I'm not sure how exactly it works, but I fixed this by changing my Run Configuration and checking the checkbox "Include dependencies with Provided scope"我不确定它是如何工作的，但我通过更改我的运行配置并选中复选框“包含具有提供范围的依赖项”来解决此问题

在此处输入图像描述

After this my SparkSQL code also works fine:在此之后，我的 SparkSQL 代码也可以正常工作：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
+---------+--------------------+-------------+-----+-------+--------+----------+
|      _c0|                 _c1|          _c2|  _c3|    _c4|     _c5|       _c6|
+---------+--------------------+-------------+-----+-------+--------+----------+
|IATA_CODE|             AIRPORT|         CITY|STATE|COUNTRY|LATITUDE| LONGITUDE|
|      ABE|Lehigh Valley Int...|    Allentown|   PA|    USA|40.65236| -75.44040|
|      ABI|Abilene Regional ...|      Abilene|   TX|    USA|32.41132| -99.68190|
|      ABQ|Albuquerque Inter...|  Albuquerque|   NM|    USA|35.04022|-106.60919|
|      ABR|Aberdeen Regional...|     Aberdeen|   SD|    USA|45.44906| -98.42183|
|      ABY|Southwest Georgia...|       Albany|   GA|    USA|31.53552| -84.19447|
|      ACK|Nantucket Memoria...|    Nantucket|   MA|    USA|41.25305| -70.06018|
|      ACT|Waco Regional Air...|         Waco|   TX|    USA|31.61129| -97.23052|
|      ACV|      Arcata Airport|Arcata/Eureka|   CA|    USA|40.97812|-124.10862|
|      ACY|Atlantic City Int...|Atlantic City|   NJ|    USA|39.45758| -74.57717|
|      ADK|        Adak Airport|         Adak|   AK|    USA|51.87796|-176.64603|
|      ADQ|      Kodiak Airport|       Kodiak|   AK|    USA|57.74997|-152.49386|
|      AEX|Alexandria Intern...|   Alexandria|   LA|    USA|31.32737| -92.54856|
|      AGS|Augusta Regional ...|      Augusta|   GA|    USA|33.36996| -81.96450|
|      AKN| King Salmon Airport|  King Salmon|   AK|    USA|58.67680|-156.64922|
|      ALB|Albany Internatio...|       Albany|   NY|    USA|42.74812| -73.80298|
|      ALO|Waterloo Regional...|     Waterloo|   IA|    USA|42.55708| -92.40034|
|      AMA|Rick Husband Amar...|     Amarillo|   TX|    USA|35.21937|-101.70593|
|      ANC|Ted Stevens Ancho...|    Anchorage|   AK|    USA|61.17432|-149.99619|
|      APN|Alpena County Reg...|       Alpena|   MI|    USA|45.07807| -83.56029|
+---------+--------------------+-------------+-----+-------+--------+----------+
only showing top 20 rows

But when I'm trying to execute Maven "clean -> package" commands, I get several errors, and all of them are regarding "org.apache" package:但是当我尝试执行 Maven “clean -> package”命令时，我收到了几个错误，所有这些都与“org.apache”package 有关：

D:\Work Projects\SparkWordCount\src\main\scala\SparkWordCount.scala:3: error: object apache is not a member of package org
import org.apache.spark.sql.{DataFrame, SparkSession}
           ^
D:\Work Projects\SparkWordCount\src\main\scala\SparkWordCount.scala:4: error: object apache is not a member of package org
import org.apache.log4j.Logger
           ^
D:\Work Projects\SparkWordCount\src\main\scala\SparkWordCount.scala:5: error: object apache is not a member of package org
import org.apache.log4j.Level
           ^
D:\Work Projects\SparkWordCount\src\main\scala\SparkWordCount.scala:10: error: not found: value Logger
    Logger.getLogger("org").setLevel(Level.ERROR)

This is details about my environment, probably some of them will be helpful:这是有关我的环境的详细信息，可能其中一些会有所帮助：

  • I use Intellij IDEa我使用 Intellij IDEa

  • Windows 10 Windows 10

  • Java 8 installed Java 8个装

在此处输入图像描述

  • Scala 2.11.8 installed and works: Scala 2.11.8 安装并工作：

在此处输入图像描述

  • Spark 2.2.0 installed and works via spark-shell Spark 2.2.0 已安装并通过 spark-shell 工作

在此处输入图像描述

  • I downloaded WinUtils, created "bin" folder in it, copy winutils.exe for hadoop_2.7.1 and paste it into "bin":我下载了 WinUtils，在其中创建了“bin”文件夹，为 hadoop_2.7.1 复制 winutils.exe 并将其粘贴到“bin”中：

在此处输入图像描述

  • This is my HADOOP_HOME and JAVA_HOME environment variables:这是我的 HADOOP_HOME 和 JAVA_HOME 环境变量：

在此处输入图像描述

  • I also set SPARK2_HOME environment variable (not sure that I actually needed to do this):我还设置了 SPARK2_HOME 环境变量（不确定我是否真的需要这样做）：

在此处输入图像描述

  • And this is my Path:这是我的路径：

在此处输入图像描述

Previoulsy I've had HADOOP_HOME in Path, but I removed it on advice from one of related threads on StackOverflow. Previoulsy 我在 Path 中有 HADOOP_HOME，但我根据 StackOverflow 上相关线程之一的建议将其删除。

Thank you in advance for your help!预先感谢您的帮助！

UPDATE-1 My project structure: UPDATE-1我的项目结构：

在此处输入图像描述

UPDATE-2更新-2

If it's important: I don't have MAVEN_HOME environment variable, and therefore there is no MAVEN_HOME in my Path.如果它很重要：我没有 MAVEN_HOME 环境变量，因此我的路径中没有 MAVEN_HOME。 I executed "clean -> package" via Intellij IDEa Maven interface.我通过 Intellij IDEa Maven 接口执行了“clean -> package”。

UPDATE-3更新-3

List of libraries from Project Structure项目结构中的库列表

在此处输入图像描述 在此处输入图像描述 在此处输入图像描述 在此处输入图像描述

Update-4 Information about scala-sdk: Update-4有关 scala-sdk 的信息： 在此处输入图像描述

1 个解决方案

解决方案1
 1  2020-05-28 09:33:32

Remove <scope>provided</scope> from pom.xml file.从 pom.xml 文件中删除<scope>provided</scope> 。 Go to file tab click on Invalidate Caches / Restart option & Try again. Go 到file选项卡单击Invalidate Caches / Restart选项并重试。

For maven issue - try mvn clean scala:compile -DdisplayCmd=true -DrecompileMode=all package command.对于 maven 问题 - 尝试mvn clean scala:compile -DdisplayCmd=true -DrecompileMode=all package命令。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 为什么当 Maven 构建运行良好但将 Spark Jar 添加为外部 Jar 时会出现编译错误“object Apache is not a member of package org” - Why when Maven Build Works good but adding Spark Jar as external Jars gives a compile error “object Apache is not a member of package org” scala -object sql不是软件包org.apache.spark的成员 - scala -object sql is not a member of package org.apache.spark Spark 2.4.3-Scala 2.12.3-对象apache不是包org的成员 - Spark 2.4.3 - Scala 2.12.3 - object apache is not a member of package org Scala error: object XML is not a member of package org.json on Apache Spark - Scala error: object XML is not a member of package org.json on Apache Spark 来自Eclipse的Spark Scala Maven构建错误-对象X不是包Y的成员 - spark scala maven build error from eclipse - Object X is not a member of package Y 在终端中运行 Scala 时出错：“对象 apache 不是包 org 的成员” - Error in running Scala in terminal: "object apache is not a member of package org" scala控制台错误：对象apache不是包org的成员 - scala console error: object apache is not a member of package org 错误：对象DataFrame不是包org.apache.spark.sql的成员 - error: object DataFrame is not a member of package org.apache.spark.sql sbt 错误：object spark 不是 package org.ZB6EFD606D118D0F62066E31ZCC19 的成员 - sbt error: object spark is not a member of package org.apache 在 Intellij 中运行 Spark 时出错：“object apache is not a member of package org” - Error in running Spark in Intellij : "object apache is not a member of package org"
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM