Python - How to prefix subprocess.run sdterror output with a time stamp?

Question

I want to prefix the sdterror output from subprocess.run with a time stamp, unfortunately I have not been about to figure out how to do so.

This part of my shell script runs FFMPEG and writes the output to a logfile :

try:
    conform_result = subprocess.run(ffmpeg_conform, stdout=PIPE, stderr=PIPE, universal_newlines=True)
    print(conform_result.stderr)
    c_log = open(config.transcode_logs + 'c_' + task_id + '_detail.txt', 'w')
    c_log.write(conform_result.stderr)
    c_log.close()
except Exception as e:
print('task ' + task_id + ' has failed for the following reason: ' + e)

I have done a lot of research into this and I can't seem to find a solution, from what I have been reading the .run is the recommended approach for running subprocess.

I know how to create the time stamp:

str(datetime.datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S"))

Can someone explain how i would prefix the time stamp to each new line from the subprocess.run() call?

EDIT:

Just to be clear I want a timestamp at the start of each line, here is what I am getting using log

Here is my logging code:

file_log = logging.getLogger()
file_log.setLevel(logging.DEBUG)
fh = logging.FileHandler(filename=task_log + 'task_' + task_id + '.txt')
formatter = logging.Formatter(fmt='%(asctime)s %(levelname)s: %(message)s',
                datefmt='%Y-%m-%d %H:%M:%S')
fh.setFormatter(formatter)
file_log.addHandler(fh)

# Conform section.
ffmpeg_conform_cmd, seg_number = functions.parse_xml(core_metadata_path, processing_temp_conform, base_mp4)
            ffmpeg_conform = str(ffmpeg_conform_cmd).replace('INPUT_FILE', conform_source)
print(timestamp() + ': ' + ffmpeg_conform)
logging.info(ffmpeg_conform)

# Updated database stating that the conform process has started
sql_conform = "UPDATE task SET status ='Conforming' WHERE task_id ='" + task_id + "'"
cursor.execute(sql_conform)
dbc.commit()
try:
   conform_result = subprocess.run(ffmpeg_conform, stdout=PIPE, stderr=PIPE, universal_newlines=True)
   print(timestamp() + ': ' +conform_result.stderr)
   file_log.info(conform_result.stderr)
except Exception as e:
   print(timestamp() + ': Conform has Failed: ' + task_id)
   print(e)
   file_log.error('Conform has Failed: ' + task_id)
   file_log.error(e)

I think the issue is that conform_result.stderr is a string and I cannot append by lines, is this the case?

BTW i am using python 3.5

Answer 1

You want to log each execution in a separate, timestamp-named file.

First, notice that it's better to avoid : in file names. Windows cannot accept that, and you want portability. So I changed the format.

Basically, it's simple:

• compute the timestamp to capture start date
• run the process
• write the logfile with the timestamp in the name

code:

try:
    timestamp = datetime.datetime.utcnow().strftime("%Y-%m-%d_%H_%M_%S")
    conform_result = subprocess.run(ffmpeg_conform, stdout=PIPE, stderr=PIPE, universal_newlines=True)
    log_file = os.path.join(config.transcode_logs,"c_{}_{}_detail.txt".format(timestamp,task_id))
    with open(log_file,"w") as c_log:
        c_log.write(conform_result.stderr)

except Exception as e:
    print('task {} has failed for the following reason: {}'.format(task_id,e))

Answer 2

这个https://docs.python.org/2/howto/logging-cookbook.html应该可以回答你的问题——它提到了来自不同进程、时间戳等的日志记录。