如何使用Apache Spark将日志写入文件

Question

I am working on a maven project. Apache spark has its own log4j functionality. Is there a way using which I can write apache spark logs and my own log statements to a file?

Answer 1

As any other java application, you need jars in classpath and you need a log4j config file.

[1] Jars in classpath

Spark already comes with these 2 in its jars folder (under $SPARK_HOME ):

slf4j-api-1.7.16.jar 
slf4j-log4j12-1.7.16.jar

So you just need a "provided" in your pom so you can compile with it:

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>1.7.16</version>
    <scope>provided</scope>
</dependency>

Note that a different version of Spark may come with a different version of slf so take a look in the [jars] folder to match the correct version to the Spark you use.

[2] A log4j config file

• Go to the conf folder under spark home dir.
• Rename this file log4j.properties.template to log4j.properties
• Add your config

Now you can add log.debug in your code.

Note that code may run on the Driver or the Executor so the logs will be in different machines depending on where it runs.