Set Apache Storm and Flink log level to display debug messages

Question

So I'm building a JAR with Storm and Flink applications where I log messages as the following:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

// ...

private static final Logger LOG = LoggerFactory.getLogger(Some.class);

// ...

LOG.debug("...");
LOG.info("...");
LOG.error("...");

Then I pass the JAR to .../bin/storm and .../bin/flink scripts and everything works, but the log level is set to INFO , I'd like to also display DEBUG messages from my package only .

I tried several things but I feel I'm just trying random things from the internet as I can't find an authoritative reference about how to obtain this and I'm having hard time wrapping my head around the incredibly confusing state of the log facilities for Java...

I'm asking about both Storm and Flink as I suspect that the root of my problem is the same but I might be wrong. Also I apologize if I don't provide a minimal example but there's really nothing to provide here.

Please let me know if you need additional details.

Answer 1

For Storm, your log configuration is in storm/log4j2/worker.xml . It's a log4j2 configuration file, so you can find out what options there are by looking at the log4j2 documentation here https://logging.apache.org/log4j/2.x/manual/configuration.html .

I'm not as familiar with Flink, but I'd suspect it's similar. Here is Flink's page on it, which mentions that you should have a logback.xml file in your conf directory.

Answer 2

In order to modify the log level and which classes log on a Flink cluster, please adapt

• FLINK_HOME/conf/log4j.properties if you are using log4j
• FLINK_HOME/conf/logback.xml if you are using logback

before you start the Flink cluster.

These files will be read when you deploy the Flink cluster. Note that these settings cannot be changed at runtime, unless you are replacing Flink's log4j logger with log4j2 which supports to load settings dynamically.

Answer 3

In this scenario:

Then I pass the JAR to .../bin/storm and .../bin/flink scripts and everything works, but the log level is set to INFO , I'd like to also display DEBUG messages from my package only .

I ended up with the following suboptimal solution.

Storm

For unknown reasons changing the /path/to/storm/log4j2/worker.xml file has no effect so I need to act programmatically:

import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.core.config.Configurator;

// ...

Configurator.setLevel("my.package", Level.ALL);

Flink

It's enough to add a line to /path/to/flink/conf/log4j.properties :

log4j.logger.my.package=ALL