Hadoop Mapreduce 作業中的 OptionConverter.convertLevel 錯誤

Question

在我的 Hadoop 集群中執行 mapreduce 作業時出現奇怪的錯誤。 此錯誤是間歇性的。 有時，它會使映射器失敗，有時即使拋出錯誤也會成功。

2022-12-28 01:20:53,882 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child: java.lang.NoSuchMethodError: org/apache/log4j/helpers/OptionConverter.convertLevel(Ljava/lang/String;Lorg/apache/logging/log4j/Level;)Lorg/apache/logging/log4j/Level; (loaded from file:/opt/hadoop-3.3.0/share/hadoop/common/lib/log4j-1.2.17.jar by sun.misc.Launcher$AppClassLoader@350ed555) called from class org.apache.log4j.config.PropertiesConfiguration (loaded from file:/data/hadoop/yarn/usercache/hdfs-user/appcache/application_1671477750397_6197/filecache/10/job.jar/job.jar by sun.misc.Launcher$AppClassLoader@350ed555).

Gradle 文件

plugins {
    id 'java'
    id "com.github.davidmc24.gradle.plugin.avro" version "1.3.0"
}
dependencies {
    compileOnly group: 'org.apache.hadoop', name: 'hadoop-client', version: '3.3.0'
    implementation group: 'org.apache.avro', name: 'avro', version: '1.8.1'
    implementation group: 'org.apache.avro', name: 'avro-mapred', version: '1.10.2'
    implementation group: 'io.netty', name: 'netty-buffer', version: '4.1.51.Final'
    implementation  group: 'joda-time', name: 'joda-time', version:'2.8.1'
    implementation group: 'org.javatuples', name: 'javatuples', version: '1.2'
    implementation group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: '2.12.4'
    testImplementation group: 'junit', name: 'junit', version: "$junit4Version"
    implementation project(':libraries:java-dcl')
    implementation project(':libraries:sdp')

    testImplementation group: 'org.assertj', name: 'assertj-core', version: '3.6.1'
}

generateAvroJava {
    source("${projectDir}/src/main/avro")//sourcepath avrofile
}

// The alternative to below is to put all java that is dependent on scala (or on the java dependent on the scala) in the src/main/scala dir
// https://stackoverflow.com/questions/23261075/compiling-scala-before-alongside-java-with-gradle
sourceSets {
    main {
        java {
            srcDirs = ['src/main/java']
        }
    }
}

Answer 1

Hadoop 不使用 log4j2，但您的 mapreduce 應用程序似乎使用了它。

您需要重新打包您的應用程序以使用正確的日志記錄庫。

Answer 2

我不得不在所有內部庫中使用 org.apache.log4j.Logger package 並擺脫其他記錄器類。 有許多文件需要更改。