[英]sed matching dates & odd formats
I am having an issue with sed where I am trying to match based on dates so I can capture all logs from a specific date/time and upload them to an API. sed出现问题,我试图根据日期进行匹配,因此我可以捕获特定日期/时间的所有日志并将其上传到API。 I then store the last run date as the new start date.
然后,我将上次运行日期存储为新的开始日期。
The problem I have is that the 'start' and 'end' dates are not necessarily present in the file and I want to match as close as I can based on the dates/times. 我的问题是文件中不一定存在“开始”和“结束”日期,并且我想根据日期/时间尽可能匹配。 The code i have at the moment it seems to only work if the two dates are present in the source file.
我目前拥有的代码似乎仅在源文件中存在两个日期的情况下才有效。
function logs() {
timestamplastupload="`cat /tmp/latest-timestamp.txt`"
timestampnow=`date +"%a %b %_d %H:%M:%S %Y"`
echo "$timestampnow" > /tmp/latest-timestamp.txt
while read -r line; do
curl -X POST -d "$line" https://logserver/api/NewLog --ntlm --user xx:xx
done < <(sed -rne '/'"$timestamplastupload"'/,/'"$timestampnow"'/ p' /var/log/fullaccess.log)
}
Is there a way to specify the sed match to do like or somehow locate the line in the file that is closest so I can ensure I am only uploading new log lines without having do a huge amount of comparison work on the API side with matching every entry in the data store there. 有没有一种方法可以指定sed匹配来进行匹配或以某种方式在文件中找到最接近的行,因此我可以确保我仅上传新的日志行,而无需在API端做大量的比较工作,每次匹配进入数据存储那里。
Here is an example of the log file I'm trying to parse: 这是我要解析的日志文件的示例:
Thu Mar 1 21:07:14 2018 us=56799 ifconfig_ipv6_pool_netbits = 0
Thu Mar 1 21:07:14 2018 us=56808 n_bcast_buf = 256
Thu Mar 1 21:07:14 2018 us=56817 tcp_queue_limit = 64
Thu Mar 1 21:07:14 2018 us=56826 real_hash_size = 256
Thu Mar 1 21:07:14 2018 us=56835 virtual_hash_size = 256
Wed Feb 28 22:10:48 2018 us=184134 ifconfig_nowarn = DISABLED
Wed Feb 28 22:10:48 2018 us=184143 ifconfig_ipv6_local = '[UNDEF]'
Wed Feb 28 22:10:48 2018 us=184152 ifconfig_ipv6_netbits = 0
Wed Feb 28 22:10:48 2018 us=184161 ifconfig_ipv6_remote = '[UNDEF]'
Also note the padded space before a single date, which might also be throwing a spanner into the works here. 还要注意单个日期之前的填充空间,这也可能使这里的扳手投入工作。 I thought I had fixed that by providing date with +%_d
我以为我已通过为日期提供+%_ d来解决此问题
Thanks in advance 提前致谢
Although sed is useful for pattern matching, it may not be suitable for value comparison. 尽管sed可用于模式匹配,但可能不适用于值比较。 AWK will be better in this sense.
从这个意义上说,AWK会更好。
A common method for time comparison is to convert the date string into seconds since the epoch. 进行时间比较的常用方法是将日期字符串转换为自纪元以来的秒数。 But it will be more practical just to merge date and time into a single number, for instance, converting "Feb 28 22:10:48 2018" into "20180228221048".
但是将日期和时间合并为一个数字会更实际,例如,将“ Feb 28 22:10:48 2018”转换为“ 20180228221048”。 Here is the example:
这是示例:
function logs() {
timestamplastupload="`cat /tmp/latest-timestamp.txt`"
timestampnow=`date +"%a %b %_d %H:%M:%S %Y"`
echo "$timestampnow" > /tmp/latest-timestamp.txt
while read -r line; do
curl -X POST -d "$line" https://logserver/api/NewLog --ntlm --user xx:xx
done < <(awk -v timestamplastupload="$timestamplastupload" -v timestampnow="$timestampnow" '
# initialize variables
BEGIN {
monstr = "JanFebMarAprMayJunJulAugSepOctNovDec";
for (i = 1; i <= 12; i++) {
mon2mm[substr(monstr, i * 3 - 2, 3)] = i;
}
split(timestamplastupload, ary, " ");
start = date2str(ary[2], ary[3], ary[4], ary[5]);
split(timestampnow, ary, " ");
end = date2str(ary[2], ary[3], ary[4], ary[5]);
}
# merge date and time into a scalar number
function date2str(mon, day, time, year,
hms, datestr) {
split(time, hms, ":");
datestr = sprintf("%04d%02d%02d%02d%02d%02d",
year, mon2mm[mon], day, hms[1], hms[2], hms[3]);
return datestr;
}
# main loop
{
logtime = date2str($2, $3, $4, $5);
if (logtime >= start && logtime <= end) {
print;
}
}
' /var/log/fullaccess.log)
}
Sorry for the lengthy and non-elegant solution. 很抱歉,冗长且不雅致的解决方案。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.