[英]problems combining files using unix cat and zcat in shell script
I've got a problem with my shell script. 我的shell脚本有问题。 I am basically trying to create a new log file from (2) files using unix (
zcat, cat
) commands 我基本上是试图使用unix(
zcat, cat
)命令从(2)个文件创建一个新的日志文件
FileA = gzipped logfile (.gz)
FileB = non-gzipped log file (.log)
FileC = new file from FileA:FileB
My problem is with FileC
. 我的问题是
FileC
。
For example: FileA has timestamp data from Aug19-Sept3
FileB has timestamp data from Sept4-Sept17
FileC has contents from both files = Aug19-Sept3:Sept4-Sept17
例如:FILEA具有从时间戳数据
Aug19-Sept3
FILEB具有从时间戳数据Sept4-Sept17
FileC具有从两个文件=内容Aug19-Sept3:Sept4-Sept17
Problem: If I manually run the commands on the server, the file looks fine as I expect FileC to have timestamp data from Aug19-Sept17
But , if done programmatically with the shell script, the local copy looks like this when I copy it back remotely: FileC = Aug19-Sept17:Aug19-Sept2
问题:如果我手动在服务器上运行的命令,文件看起来很好,我希望FileC有从时间戳数据
Aug19-Sept17
但是 ,如果与shell脚本编程完成后,本地副本是这样的,当我将它复制回远程:FileC = Aug19-Sept17:Aug19-Sept2
So my question is why is this happening programmatically? 所以我的问题是为什么这会以编程方式发生? Is there a better way to combine/stitch these 2 logs together to create another one?
有没有更好的方法将这两个日志组合/缝合在一起以创建另一个日志?
Here is my shell script: 这是我的shell脚本:
#!/bin/bash
if [ $# != 1 ]; then
echo "Usage: getlogs.sh <remote-host>" 2>&1
exit 1
fi
#Declare variables
STAMP=`date '+%Y%m%d-%H:%M'`
REMOTE_MYCNF=/var/log/mysoft/mysoft.log
REMOTE_GZ=/var/log/mysoft/mysoft.log.1.gz
REMOTE_DIR=/var/log/mysoft/
BACKUP_DIR=/home/dev/logs/
NEWLOG="foo-temp.log"
#Copy file over
echo "START..." 2>&1
test -f $BACKUP_DIR$1.mysoft.log
if [ $? = 0 ]; then
echo "Local log file $BACKUP_DIR$1.mysoft.log exists, clean up for new copy..." 2>&1
/bin/rm $BACKUP_DIR$1.mysoft.log
else
echo "File does not exist, getting a new copy..." 2>&1
fi
echo "Checking remotely in $1 for logfile $REMOTE_MYCNF $STAMP" 2>&1
if [ ! -f $REMOTE_MYCNF ]; then
echo "File exists remotely, creating new logfile and copy here...." 2>&1
ssh $1 "zcat $REMOTE_GZ >> $REMOTE_DIR$NEWLOG"
ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"
/usr/bin/scp $1:$REMOTE_DIR$NEWLOG $BACKUP_DIR$1.mysoft.log
echo "end remote copy" 2>&1
echo "Cleaning up remote files" 2>&1
ssh $1 "rm $REMOTE_DIR$NEWLOG"
exit 0
else
echo "Unable to get file" 2>&1
exit 0
fi
Perhaps it's possible that the cat
is starting before the zcat
is finished. 也许
cat
可能在zcat
完成之前就开始了。 However, it would surprise me if this were the case. 但是,如果是这样的话,会让我感到惊讶。
Try this: 尝试这个:
ssh $1 "zcat ... >> ...; cat ... >> ..."
or 要么
ssh $1 "zcat ... >> ..."
sleep 5
ssh $1 "cat ... >> ..."
or 要么
ssh $1 "zcat -f file1 file2 >> ..." # let zcat do both files
I can't guess immediately what the error is, but have you tried simplifying your shell script? 我无法立即猜测出错误是什么,但是您是否尝试简化shell脚本? That is, start with this:
也就是说,从以下开始:
#!/bin/sh
STAMP=`date '+%Y%m%d-%H:%M'`
REMOTE_MYCNF=/var/log/mysoft/mysoft.log
REMOTE_GZ=/var/log/mysoft/mysoft.log.1.gz
REMOTE_DIR=/var/log/mysoft/
BACKUP_DIR=/home/dev/logs/
NEWLOG="foo-temp.log"
ssh $1 "zcat $REMOTE_GZ >> $REMOTE_DIR$NEWLOG"
ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"
Then build up from here. 然后从这里开始。 This will at least allow you to isolate the problem.
这至少可以让您隔离问题。
You don't need to create a remote temporary file at all. 您根本不需要创建远程临时文件。 Get rid of the multiple ssh commands, scp, and cleanup ssh, and try this instead:
摆脱多个ssh命令,scp和cleanup ssh,然后尝试以下操作:
ssh $1 "zcat $REMOTE_GZ; cat $REMOTE_MYCNF;" > $BACKUP_DIR$1.mysoft.log
zcat and cat will write data to their stdout, which is attached to ssh's stdout, which we redirect to a file. zcat和cat会将数据写入其stdout,该stdout附加到ssh的stdout,我们将其重定向到文件。
I "guarantee" that zcat writes all its data and quits before cat begins, giving the desired ordering. 我“保证” zcat写入其所有数据并在cat开始之前退出,并给出所需的顺序。 "guarantee" is in quotes because the other solutions should work, which are reported not to.
用“保证”作为引号,因为其他解决方案应该起作用,但据报告不起作用。
It seems the temp file is getting reused, delete or truncate it: 临时文件似乎正在被重用,删除或截断:
# Always using the same log for new log.
NEWLOG="foo-temp.log"
# Use a new log truncate '>' the old one
ssh $1 "zcat $REMOTE_GZ > $REMOTE_DIR$NEWLOG"
# This time, append '>>' to new log
ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.