Disable logging from a specific class in Apache Beam running on Google Cloud Dataflow

Question

I saw on this website https://cloud.google.com/dataflow/docs/guides/logging that i can programmatically set logging behavior of the pipeline. In my case, there is a class from a third party dependency that is producing too much warning messages, so i want to set its logging level to ERROR only.

So what i did: (a bit different than what the website provided (out of date))

val sdkHarnessLogLevelOverrides = SdkHarnessLogLevelOverrides()
sdkHarnessLogLevelOverrides
    .addOverrideForClass(
        ThirdPartyClass::class.java,
        SdkHarnessOptions.LogLevel.ERROR
    )

val optionsWithLoggingConfiguration = options.`as`(SdkHarnessOptions::class.java)
optionsWithLoggingConfiguration.sdkHarnessLogLevelOverrides = loggingConfiguration
//I also want to set the default logging behavior for the rest to WARN level 
optionsWithLoggingConfiguration.defaultSdkHarnessLogLevel = SdkHarnessOptions.LogLevel.WARN

val pipeline = Pipeline.create(optionsWithLoggingConfiguration)

But I still can see the WARN message from the ThirdPartyClass in the Stackdrive, which coming from dataflow_step in a dataflow worker.

I don't know what I did wrong here, the documentation is pretty out of dated and limited examples. Does anyone encounter this problem before? I would love to hear from you. Thank you very much.

Answer 1

Try using DataflowWorkerLoggingOptions.setWorkerLogLevelOverrides

Explanation: Dataflow currently can run in two different modes, which we call "v1" and "v2":

• In Dataflow v1 Java pipelines run their DoFn code directly in the same process as the dataflow worker, so you use DataflowWorkerLoggingOptions.setWorkerLogLevelOverrides .
• In Dataflow v2, which you can activate by --experiments=use_runner_v2 and will eventually become the default, the SDK harness runs your DoFn code in a separate process from the highly optimized C++-based Dataflow v2 worker, so you use SdkHarnessOptions.setSdkHarnessLogLevelOverrides

A bonus about the v2 approach is that this works on all modernized Beam runners, not just Dataflow, as the SDK harness that runs your DoFn is a portable process that is the same regardless of runner that you choose.