Scala：線程“ main”中的異常java.lang.NoClassDefFoundError：org / apache / log4j / LogManager

Question

我是Scala的初學者，我正在嘗試在Scala中運行模型，但面臨一些問題：

這是文件：

package com.salesforce.hw.titanic

import com.salesforce.op._
import com.salesforce.op.features.FeatureBuilder
import com.salesforce.op.features.types._
import com.salesforce.op.readers.DataReaders
import com.salesforce.op.stages.impl.classification._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Level, LogManager}



/**
 * A minimal Titanic Survival example with TransmogrifAI
 */



object OpTitanicMini {

  case class Passenger
  (
    id: Long,
    survived: Double,
    pClass: Option[Long],
    name: Option[String],
    sex: Option[String],
    age: Option[Double],
    sibSp: Option[Long],
    parCh: Option[Long],
    ticket: Option[String],
    fare: Option[Double],
    cabin: Option[String],
    embarked: Option[String]
  )



  def main(args: Array[String]): Unit = {
    LogManager.getLogger("com.salesforce.op").setLevel(Level.ERROR)
    implicit val spark = SparkSession.builder.config(new SparkConf()).getOrCreate()
    import spark.implicits._

    // Read Titanic data as a DataFrame
    val pathToData = Option(args(0))
    val passengersData = DataReaders.Simple.csvCase[Passenger](pathToData, key = _.id.toString).readDataset().toDF()

    // Automated feature engineering
    val (survived, features) = FeatureBuilder.fromDataFrame[RealNN](passengersData, response = "survived")
    val featureVector = features.toSeq.autoTransform()

    // Automated feature selection
    val checkedFeatures = survived.sanityCheck(featureVector, checkSample = 1.0, removeBadFeatures = true)

    // Automated model selection
    val (pred, raw, prob) = BinaryClassificationModelSelector().setInput(survived, checkedFeatures).getOutput()
    val model = new OpWorkflow().setInputDataset(passengersData).setResultFeatures(pred).train()

    println("Model summary:\n" + model.summaryPretty())








  }

}

當我嘗試運行它時，出現此錯誤：

 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/LogManager at com.salesforce.hw.titanic.OpTitanicMini$.main(OpTitanicMini.scala:72) at com.salesforce.hw.titanic.OpTitanicMini.main(OpTitanicMini.scala) Caused by: java.lang.ClassNotFoundException: org.apache.log4j.LogManager at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 2 more 

我嘗試查看此問題並找到了該博客文章 ，我嘗試了該博客文章所說的內容：

我的log4j.properties文件如下所示：

log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.Remoting=ERROR
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# Change this to set Hadoop log level
log4j.logger.org.apache.hadoop=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

# Breeze
log4j.logger.breeze.optimize=FATAL

# BLAS & LAPACK
log4j.logger.com.github.fommil.netlib=ERROR

# TransmogrifAI logging
log4j.logger.com.salesforce.op=INFO
log4j.logger.com.salesforce.op.utils.spark.OpSparkListener=OFF

# Helloworld logging
log4j.logger.com.salesforce.hw=INFO

我嘗試了博客文章中提到的步驟，但是仍然遇到相同的問題，我該如何解決這個問題？

Answer 1

LogManager類帶有Spark依賴項之一。 確保在運行時具有org.apache.spark:spark-core ， org.apache.spark:spark-mlib ， org.apache.spark:spark-sql及其所有可傳遞依賴項。

我們在這里有一個示例sbt項目，您可以看一下。