problems combining files using unix cat and zcat in shell script

Question

I've got a problem with my shell script. I am basically trying to create a new log file from (2) files using unix ( zcat, cat ) commands

FileA = gzipped logfile (.gz)
FileB = non-gzipped log file (.log)

FileC = new file from FileA:FileB

My problem is with FileC .

For example: FileA has timestamp data from Aug19-Sept3 FileB has timestamp data from Sept4-Sept17 FileC has contents from both files = Aug19-Sept3:Sept4-Sept17

Problem: If I manually run the commands on the server, the file looks fine as I expect FileC to have timestamp data from Aug19-Sept17 But , if done programmatically with the shell script, the local copy looks like this when I copy it back remotely: FileC = Aug19-Sept17:Aug19-Sept2

So my question is why is this happening programmatically? Is there a better way to combine/stitch these 2 logs together to create another one?

Here is my shell script:

#!/bin/bash
if [ $# != 1 ]; then
    echo "Usage: getlogs.sh <remote-host>" 2>&1
    exit 1
fi

#Declare variables
STAMP=`date '+%Y%m%d-%H:%M'`
REMOTE_MYCNF=/var/log/mysoft/mysoft.log
REMOTE_GZ=/var/log/mysoft/mysoft.log.1.gz
REMOTE_DIR=/var/log/mysoft/
BACKUP_DIR=/home/dev/logs/
NEWLOG="foo-temp.log"

#Copy file over
echo "START..." 2>&1
test -f $BACKUP_DIR$1.mysoft.log
if [ $? = 0 ]; then
   echo "Local log file $BACKUP_DIR$1.mysoft.log exists, clean up for new copy..." 2>&1
   /bin/rm $BACKUP_DIR$1.mysoft.log
   else
        echo "File does not exist, getting a new copy..." 2>&1
fi

echo "Checking remotely in $1 for logfile $REMOTE_MYCNF $STAMP" 2>&1
if [ ! -f $REMOTE_MYCNF ]; then
   echo "File exists remotely, creating new logfile and copy here...." 2>&1
   ssh $1 "zcat $REMOTE_GZ >> $REMOTE_DIR$NEWLOG"
   ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"
   /usr/bin/scp $1:$REMOTE_DIR$NEWLOG $BACKUP_DIR$1.mysoft.log
   echo "end remote copy" 2>&1
   echo "Cleaning up remote files" 2>&1
   ssh $1 "rm $REMOTE_DIR$NEWLOG"
   exit 0
   else
        echo "Unable to get file" 2>&1
        exit 0
fi

Answer 1

Perhaps it's possible that the cat is starting before the zcat is finished. However, it would surprise me if this were the case.

Try this:

ssh $1 "zcat ... >> ...; cat ... >> ..."

or

ssh $1 "zcat ... >> ..."
sleep 5
ssh $1 "cat ... >> ..."

or

ssh $1 "zcat -f file1 file2 >> ..."   # let zcat do both files

Answer 2

I can't guess immediately what the error is, but have you tried simplifying your shell script? That is, start with this:

#!/bin/sh
STAMP=`date '+%Y%m%d-%H:%M'`
REMOTE_MYCNF=/var/log/mysoft/mysoft.log
REMOTE_GZ=/var/log/mysoft/mysoft.log.1.gz
REMOTE_DIR=/var/log/mysoft/
BACKUP_DIR=/home/dev/logs/
NEWLOG="foo-temp.log"
ssh $1 "zcat $REMOTE_GZ >> $REMOTE_DIR$NEWLOG"
ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"

Then build up from here. This will at least allow you to isolate the problem.

Answer 3

You don't need to create a remote temporary file at all. Get rid of the multiple ssh commands, scp, and cleanup ssh, and try this instead:

ssh $1 "zcat $REMOTE_GZ; cat $REMOTE_MYCNF;" > $BACKUP_DIR$1.mysoft.log

zcat and cat will write data to their stdout, which is attached to ssh's stdout, which we redirect to a file.

I "guarantee" that zcat writes all its data and quits before cat begins, giving the desired ordering. "guarantee" is in quotes because the other solutions should work, which are reported not to.

Answer 4

It seems the temp file is getting reused, delete or truncate it:

# Always using the same log for new log.
NEWLOG="foo-temp.log"
# Use a new log truncate '>' the old one
ssh $1 "zcat $REMOTE_GZ > $REMOTE_DIR$NEWLOG"
# This time, append '>>' to new log
ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"