如何在单元测试中抑制Spark记录？

Question

So thanks to easily googleable blogs I tried:

import org.specs2.mutable.Specification

class SparkEngineSpecs extends Specification {
  sequential

  def setLogLevels(level: Level, loggers: Seq[String]): Map[String, Level] = loggers.map(loggerName => {
    val logger = Logger.getLogger(loggerName)
    val prevLevel = logger.getLevel
    logger.setLevel(level)
    loggerName -> prevLevel
  }).toMap

  setLogLevels(Level.WARN, Seq("spark", "org.eclipse.jetty", "akka"))

  val sc = new SparkContext(new SparkConf().setMaster("local").setAppName("Test Spark Engine"))

  // ... my unit tests

But unfortunately it doesn't work, I still get a lot of spark output, eg:

14/12/02 12:01:56 INFO MemoryStore: Block broadcast_4 of size 4184 dropped from memory (free 583461216)
14/12/02 12:01:56 INFO ContextCleaner: Cleaned broadcast 4
14/12/02 12:01:56 INFO ContextCleaner: Cleaned shuffle 4
14/12/02 12:01:56 INFO ShuffleBlockManager: Deleted all files for shuffle 4

Answer 1

Add the following code into the log4j.properties file inside the src/test/resources dir, create the file/dir if not exist

# Change this to set Spark log level
log4j.logger.org.apache.spark=WARN

# Silence akka remoting
log4j.logger.Remoting=WARN

# Ignore messages below warning level from Jetty, because it's a bit verbose
log4j.logger.org.eclipse.jetty=WARN

When I run my unit tests (I'm using JUnit and Maven), I only receive WARN level logs, in other words no more cluttering with INFO level logs (though they can be useful at times for debugging).

I hope this helps.

Answer 2

In my case one of my own libraries brought logback-classic into the mix. This materialized in a warning at the start:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

I solved this by excluding it from the dependency:

"com.mystuff" % "mylib" % "1.0.0" exclude("ch.qos.logback", "logback-classic")

Now I could add a log4j.properties file in test/resources which now gets used by Spark.

Answer 3

After some time of struggling with Spark log output as well, I found a blog post with a solution I particularly liked.

If you use slf4j, one can simply exchange the underlying log implementation. A good canidate for the test scope is slf4j-nop, which carfully takes the log output and puts it where the sun never shines.

When using Maven you can add the following to the top of your dependencies list:

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>1.7.12</version>
  <scope>provided</scope>
</dependency>

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-nop</artifactId>
  <version>1.7.12</version>
  <scope>test</scope>
</dependency>

Note that it might be important to have it at the beginning of the dependencies list to make sure that the given implementations are used instead of those that might come with other packages (and which you can consider to exclude in order to keep your class path tidy and avoid unexpected conflicts).

Answer 4

You can use a separate Logback config for tests. Depending on your environment it's possible that you just need to create conf/logback-test.xml with something that hides the logs. I think this should do that:

<configuration>
  <root level="debug">
  </root>
</configuration>

As I understand it, this captures all logs (level debug and higher) and assigns no logger to them, so they get discarded. A better option is to configure a file logger for them, so you can still access the logs if you want to.

See http://logback.qos.ch/manual/configuration.html for the detailed documentation.

Answer 5

A little late to the party but I found this in the spark example code :

def setStreamingLogLevels() {
    val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements
    if (!log4jInitialized) {
        // We first log something to initialize Spark's default logging, then we override the
        // logging level.
        logInfo("Setting log level to [WARN] for streaming example." +
        " To override add a custom log4j.properties to the classpath.")
        Logger.getRootLogger.setLevel(Level.WARN)
    }
}

I also found that with your code if you call setLogLevels like below it cut out alot of out put for me.

setLogLevels(Level.WARN, Seq("spark", "org", "akka"))

Answer 6

The easiest solution working for me is:

cp $SPARK_HOME/conf/log4j.properties.template $YOUR_PROJECT/src/test/resources/log4j.properties
sed -i -e 's/log4j.rootCategory=INFO/log4j.rootCategory=WARN/g' $YOUR_PROJECT/src/test/resources/log4j.properties