如何gzip awk命令的结果输出？

Question

I tried this but it did not work. How can I generate the output to be a gz file from an awk command?

 awk -v someVariable="$VAR1" '
        BEGIN {
         . . .
        }

        {
           SOME CODES HERE
        }
        END {}
    ' $FILES> gzip $RESULTING_OUTPUT

Answer 1

You can EITHER pipe your output to gzip outside of awk (as Tom suggested), or you can send output through pipes from inside awk. Like this:

awk '
  {
    print | "gzip > /path/to/output.gz";
  }
' inputfile

This has a tremendous advantage in certain situations. For example, if you want to split a really long log file into hourly chunks, you can build your output command as a variable, then pipe to the variable.

awk '

  # given:
  #   Mar 20 13:29:12 servername some message

  BEGIN {
    m["jan"]="01"; m["feb"]="02"; m["mar"]="03"; m["apr"]="04"; 
    m["may"]="05"; m["jun"]="06"; m["jul"]="07"; m["aug"]="08"; 
    m["sep"]="09"; m["oct"]="10"; m["nov"]="11"; m["dec"]="12";
  }

  {
    output=sprintf("gzip -9 > /var/log/split/%s-%s-%s.log.gz", m[tolower($1)], $2, substr($3,1,2));
    print | output
  }' input.log

With this usage, your output gzip command line is re-evaluated for every line of input, and awk doesn't close the pipe unless it's told to manually, or awk runs out of input and exits.

My own use case for this was that we were gathering web server logs from a CDN that were not in chronological order. The logs were way too large for sort , but could be handled when split into hourly chunks.

YMMV. The best solution depends on what you're actually trying to achieve, which you haven't told us.

Answer 2

You need to pipe the output to gzip , then redirect then output to a file:

awk '...' $FILES | gzip > "$RESULTING_OUTPUT"

Note that capital letters for variables names are not recommended, as they may clash with shell internal variables. Also, $FILES looks suspiciously like it may contain a list of more than one file name. You should really be using an array, which you can pass like "${files[@]}"

Answer 3

This worked

 awk -v someVariable="$VAR1" '
            BEGIN {
             . . .
            }

            {
               SOME CODES HERE
            }
            END {}
        ' $FILES> $RESULTING_OUTPUT
    gzip $RESULTING_OUTPUT