systemd service for hadoop start-dfs.sh

Question

I am having trouble getting Hadoop dfs running as a system service on Ubuntu18.04. Running start-dfs.sh from the terminal works fine, but running as a systemd service fails.

I also tried the example here ( Systemd Hdfs Service [hadoop] - startup ) but it fails.

The service file..

[Unit]
Description=Hadoop DFS namenode and datanode
After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target
Requires=network-online.target

[Service]
User=ubuntu
Group=ubuntu
Type=forking
ExecStart=/opt/hadoop/sbin/start-dfs.sh
ExecStop=/opt/hadoop/sbin/stop-dfs.sh
WorkingDirectory=/home/ubuntu
Environment=JAVA_HOME=/usr/lib/jvm/jdk1.8.0_251
Environment=SPARK_HOME=/opt/spark
Environment=HADOOP_HOME=/opt/hadoop
Environment=HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
Environment=HADOOP_OPTS="-Djava.library.path=/opt/hadoop/lib/native"
Environment=YARN_HOME=/opt/hadoop
Environment=LD_LIBRARY_PATH=/opt/hadoop/lib/native
TimeoutStartSec=2min
#Restart=on-failure
PIDFile=/tmp/hadoop-ubuntu-namenode.pid
RestartSec= 30
StartLimitInterval=350
StartLimitBurst=10
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

The error...

ubuntu@sc:/etc/systemd/system$ sudo service hadoop-dfs status
● hadoop-dfs.service - Hadoop DFS namenode and datanode
   Loaded: loaded (/etc/systemd/system/hadoop-dfs.service; disabled; vendor preset: enabled)
   Active: failed (Result: protocol) since Wed 2020-10-14 14:28:19 UTC; 8s ago
  Process: 4992 ExecStart=/opt/hadoop/sbin/start-dfs.sh (code=exited, status=0/SUCCESS)

Oct 14 14:28:00 sc systemd[1]: Starting Hadoop DFS namenode and datanode...
Oct 14 14:28:01 sc start-dfs.sh[4992]: Starting namenodes on [localhost]
Oct 14 14:28:06 sc start-dfs.sh[4992]: localhost: starting namenode, logging to /opt/hadoop-2.8.5/logs/hadoop-ubuntu-namenode-sc.out
Oct 14 14:28:11 sc start-dfs.sh[4992]: localhost: starting datanode, logging to /opt/hadoop-2.8.5/logs/hadoop-ubuntu-datanode-sc.out
Oct 14 14:28:12 sc start-dfs.sh[4992]: Starting secondary namenodes [0.0.0.0]
Oct 14 14:28:17 sc start-dfs.sh[4992]: 0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.8.5/logs/hadoop-ubuntu-secondarynamenode-sc.out
Oct 14 14:28:19 sc systemd[1]: hadoop-dfs.service: New main PID 6948 does not belong to service, and PID file is not owned by root. Refusing.
Oct 14 14:28:19 sc systemd[1]: hadoop-dfs.service: Failed with result 'protocol'.
Oct 14 14:28:19 sc systemd[1]: Failed to start Hadoop DFS namenode and datanode.

---

After setting up a dedicated user and running just the namenode

● hadoop-dfs.service - Hadoop DFS namenode and datanode
   Loaded: loaded (/etc/systemd/system/hadoop-dfs.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-10-14 15:59:35 UTC; 5s ago
  Process: 12633 ExecStart=/opt/hadoop/sbin/hadoop-daemon.sh start namenode (code=exited, status=1/FAILURE)

Oct 14 15:59:31 sc systemd[1]: Starting Hadoop DFS namenode and datanode...
Oct 14 15:59:31 sc hadoop-daemon.sh[12633]: starting namenode, logging to /opt/hadoop-2.8.5/logs/hadoop-hadoop-namenode-sc.out
Oct 14 15:59:35 sc systemd[1]: hadoop-dfs.service: Control process exited, code=exited status=1
Oct 14 15:59:35 sc systemd[1]: hadoop-dfs.service: Failed with result 'exit-code'.
Oct 14 15:59:35 sc systemd[1]: Failed to start Hadoop DFS namenode and datanode.

Answer 1

Try this service definition. It worked for me.

[Unit]
Description=Hadoop DFS namenode and datanode
After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target
Requires=network-online.target

[Service]
User=hadoop
Group=hadoop
Type=forking
ExecStart=/app/hadoop/sbin/start-dfs.sh
ExecStop=/app/hadoop/sbin/stop-dfs.sh
WorkingDirectory=/app/hadoop
TimeoutStartSec=1min
# Restart=on-failure
Restart=no
RestartSec=30
RemainAfterExit=yes
# PIDFile=/tmp/hadoop-hadoop-namenode.pid

[Install]
WantedBy=multi-user.target