Linux shell脚本将1个月以上的tar.gzip日志文件按月分组

Question

I have a directory full of various application logs. Example:

FailedAudit_20150101_000000.log FailedAudit_20150209_000000.log FailedAudit_20150316_000000.log stats20150116.log stats20150224.log FailedAudit_20150102_000000.log FailedAudit_20150210_000000.log FailedAudit_20150317_000000.log stats20150117.log stats20150225.log FailedAudit_20150103_000000.log RepoV4Error20150227.log

All the logs have timestamp in format YYYYMMDD but also other numbers involved as you can see.

My objective is to write a script that can be run once periodically to go through this directory and do the following: For all log files older than 1 month, based on filename timestamp

• for each months worth of files (30~31 files), tar.gz them into one file
• label the tar.gz file as

App1_201508.tar.gz <-- contains all 30 log files So format AppnameYYYYMM.tar.gz

The log file application name is static except for the timestamp.

I suppose there is a few ways to do this but I would like to gather ideas from the great minds of stackoverflow to find the simplest way.

Thanks in advance

Answer 1

Here's the third solution for your updated question:

#!/usr/bin/env bash

LOGTYPES=$( ls *log* | sed -rn "s/([0-9]{6})[0-9]{2}.*$/\1/p" | sort -u )

# the sed command, item by item:
#
# s/ search and replace
# ([0-9]{6}) block of 6 digits, and store it
# [0-9]{2} followed by 2 more digits
# .*$ followed by any and all characters until the end of the input
# / replace all of that with
# \1 the first stored block (the 6 digits)
# /p print the output
#
# So this turns FailedAudit_20150101_000000.log into FailedAudit_201501

THIS_MONTH=$(date +%Y%m)
for LOG in $LOGTYPES; do
    MONTH=${LOG: -6} # Last 6 characters of the LOGTYPE are YYYYMM

    if [[ "$MONTH" -lt "$THIS_MONTH" ]]; then
        LOG_FILES=$(ls ${LOG}*)
        tar -czf ${LOG}.tar.gz ${LOG_FILES}
        RC=$? # Check whether an error occured
        if [[ "$RC" == "0" ]]; then
            rm ${LOG_FILES}
        fi
    fi
done

Note: This assumes that the first block of 8 digits is the datestamp, and everything after that is not relevant for which archive it is to go to.

Update: The sed script no longer outputs files that do not contain a timestamp.

Answer 2

here, not sure if workimg

#!/bin/bash
MONTH=$(date +%m)
OLDMONTH=$MONTH-1
for FILE in `ls $DIR`
do
    if [ ${FILE:-4:2} == $OLDMONTH]; then
        # do what you want with the file, it's one month old, eg add it to a list
    fi
done
# do what you want with the list, eg tar,... 

run the script once a day as example with runwhen or cron