错误：无法找到或加载主类 org.apache.tez.dag.app.DAGAppMaster

Question

I have installed tez and want to run the example like this

hadoop jar tez-examples-0.10.1.jar orderedwordcount /input /output

but it's not work and the log is

Log Type: stderr

Log Upload Time: Thu May 12 13:19:25 +0800 2022

Log Length: 77

Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster

Log Type: stdout

Log Upload Time: Thu May 12 13:19:25 +0800 2022

Log Length: 716

Heap
 PSYoungGen      total 17920K, used 921K [0x00000000eef00000, 0x00000000f0300000, 0x0000000100000000)
  eden space 15360K, 6% used [0x00000000eef00000,0x00000000eefe67a8,0x00000000efe00000)
  from space 2560K, 0% used [0x00000000f0080000,0x00000000f0080000,0x00000000f0300000)
  to   space 2560K, 0% used [0x00000000efe00000,0x00000000efe00000,0x00000000f0080000)
 ParOldGen       total 40960K, used 0K [0x00000000ccc00000, 0x00000000cf400000, 0x00000000eef00000)
  object space 40960K, 0% used [0x00000000ccc00000,0x00000000ccc00000,0x00000000cf400000)
 Metaspace       used 2541K, capacity 4480K, committed 4480K, reserved 1056768K
  class space    used 283K, capacity 384K, committed 384K, reserved 1048576K

my_env.sh is

#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_331

export PATH=$PATH:$JAVA_HOME/bin

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.3.2
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

#HIVE_HOME
export HIVE_HOME=/opt/module/hive-3.1.3
export PATH=$PATH:$HIVE_HOME/bin

#MAVEN_HOME
export MAVEN_HOME=/opt/module/maven-3.8.5
export PATH=$PATH:$MAVEN_HOME/bin

#TEZ_HOME
export TEZ_HOME=/opt/module/tez-0.10.1
export HADOOP_CLASSPATH=${TEZ_HOME}/conf:${TEZ_HOME}/*:${TEZ_HOME}/lib/*

tez-site.xml is

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>tez.lib.uris</name>
    <value>${fs.defaultFS}/apps/tez/tez-0.10.1.tar.gz</value>
</property>
<property>
    <name>tez.use.cluster.hadoop-libs</name>
    <value>false</value>
</property>
<property>
    <name>tez.history.logging.service.class</name>
    <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>
</configuration>

I have tried answer https://issues.apache.org/jira/browse/TEZ-3392 but it's not work. Please help me in resolving this.Thanks in Advance!!!

Answer 1

I had the similar error message with running tez example. The installation guide https://tez.apache.org/install.html is not so obious, but has some valueable notes.

Helpful was an application tracking page where from logs i found out that path in the container after decompress tez archive is incorrect. The whole archive is decompressed to the directory called ./tezlib and excerpt of CLASSPATH looks like that:

$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*

but archive apache-tez-0.10.1-bin.tar.gz (on my HDFS in path /apps/apache-tez-0.10.1-bin.tar.gz) is decompressed inside a container to ./tezlib/apache-tez-0.10.1-bin.

So, after several hours trial and error i resolved this issue in the following steps:

tar -xf apache-tez-0.10.1-bin.tar.gz
tar -czf apache-tez-0.10.1-bin-nodir.tar.gz -C apache-tez-0.10.1-bin .
hdfs dfs -copyFromLocal apache-tez-0.10.1-bin-nodir.tar.gz /apps/

The second line above pack tez jars into an archive without parent directory. After that tez example runs without error and finishes succeed.

My tez-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/apache-tez-0.10.1-bin-nodir.tar.gz</value>
</property>

<property>
<name>tez.use.cluster.hadoop-libs</name>
<value>true</value>
</property>
</configuration>

Of course there are propably another ways to manage this error with incorrect path to jars.

I've tested that at hadoop 3.2.2 from bigtop distribution and tez 0.10.0/0.10.1.

Answer 2

I had solved my question by upload to hdfs an uncompressed tez package and change my tez-site.xml file.

hadoop fs -put tez-0.10.1 /apps/tez

My changed tez-site.xml

The main different place is "tez.lib.uris"

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>tez.lib.uris</name>
        <value>${fs.defaultFS}/apps/tez/tez-0.10.1,${fs.defaultFS}/apps/tez/tez-0.10.1/lib</value>
    </property>
    <property>
        <name>tez.use.cluster.hadoop-libs</name>
        <value>false</value>
    </property>
    <property>
        <name>tez.history.logging.service.class</name>
        <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
    </property>
    <property>
        <name>tez.am.resource.memory.mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>tez.am.resource.cpu.vcores</name>
        <value>1</value>
    </property>
</configuration>