如何在HDFS中存储Shell脚本的日志文件

Question

I have shell script in HDFS . I want to collect the logs for this script in HDFS only.

The contents of the script are below:

#!/bin/bash

TIMESTAMP=`date "+%Y-%m-%d"`

hdfs dfs -touchz /user/$USER/logs/`date "+%Y-%m-%d"`/${TIMESTAMP}.success_log

hdfs dfs -touchz /user/$USER/logs/`date "+%Y-%m-%d"`/${TIMESTAMP}.fail_log

success_logs=/user/$USER/logs/`date "+%Y-%m-%d"`/${TIMESTAMP}.success_log

failed_logs=/user/$USER/logs/`date "+%Y-%m-%d"`/${TIMESTAMP}.fail_log


function log_status
{
   status=$1
   message=$2
   if [ "$status" -ne 0 ]; then
            echo "`date +\"%Y-%m-%d %H:%M:%S\"` [ERROR] $message [Status] $status : failed" | tee -a "${failed_logs}"
            #echo "Please find the attached log file for more details"
            exit 1
            else
                echo "`date +\"%Y-%m-%d %H:%M:%S\"` [INFO] $message [Status] $status : success" | tee -a "${success_logs}"
            fi
}

The logs are not appending to the files. only the files are being created.

How can I get the files to have the result of the function to be appended in HDFS

Answer 1

The logs are not appending to the files. only the files are being created.

Because tee is a linux command and does not work for files stored in HDFS.

Use -appendToFile

echo "`date +\"%Y-%m-%d %H:%M:%S\"` [ERROR] $message [Status] $status : failed" | hdfs dfs -appendToFile - ${failed_logs}

- in place of srcfile is to read the input from stdin.