合并排序gzip压缩文件

Question

I have 40 files of 2GB each, stored on an NFS architecture. Each file contains two columns: a numeric id and a text field. Each file is already sorted and gzipped.

How can I merge all of these files so that the resulting output is also sorted?

I know sort -m -k 1 should do the trick for uncompressed files, but I don't know how to do it directly with the compressed ones.

PS: I don't want the simple solution of uncompressing the files into disk, merging them, and compressing again, as I don't have sufficient disk space for that.

Answer 1

This is a use case for process substitution . Say you have two files to sort, sorta.gz and sortb.gz . You can give the output of gunzip -c FILE.gz to sort for both of these files using the <(...) shell operator:

sort -m -k1 <(gunzip -c sorta.gz) <(gunzip -c sortb.gz) >sorted

Process substitution substitutes a command with a file name that represents the output of that command, and is typically implemented with either a named pipe or a /dev/fd/... special file.

For 40 files, you will want to create the command with that many process substitutions dynamically, and use eval to execute it:

cmd="sort -m -k1 "
for input in file1.gz file2.gz file3.gz ...; do
    cmd="$cmd <(gunzip -c '$input')"
done
eval "$cmd" >sorted       # or eval "$cmd" | gzip -c > sorted.gz

Answer 2

    #!/bin/bash

    FILES=file*.gz               # list of your 40 gzip files
                                 # (e.g. file1.gz ... file40.gz)

    WORK1="merged.gz"            # first temp file and the final file
    WORK2="tempfile.gz"          # second temp file

    > "$WORK1"                   # create empty final file
    > "$WORK2"                   # create empty temp file

    gzip -qc "$WORK2" > "$WORK1" # compress content of empty second
                                 # file to first temp file

    for I in $FILES; do
        echo current file: "$I"
        sort -k 1 -m <(gunzip -c "$I") <(gunzip -c "$WORK1") | gzip -c > "$WORK2"
        mv "$WORK2" "$WORK1"
    done

Fill $FILES the easiest way with the list of your files with bash globbing (file*.gz) or with a list of 40 filenames (separated with white blanks). Your files in $FILES stay unchanged.

Finally, the 80 GB data are compressed in $WORK1. While processing this script no uncompressed data where written to disk.

Answer 3

Adding a differently flavoured multi-file merge within a single pipeline - it takes all (pre-sorted) files in $OUT/uniques , sort-merges them and compresses the output, lz4 is used due to it's speed:

find $OUT/uniques -name '*.lz4' |
  awk '{print "<( <" $0 " lz4cat )"}' |
  tr "\n" " " |
  (echo -n sort -m -k3b -k2 " "; cat -; echo) |
  bash |
  lz4 \
> $OUT/uniques-merged.tsv.lz4

Answer 4

确实有zgrep和其他常用实用程序可以使用压缩文件，但在这种情况下，您需要对未压缩数据进行排序/合并并压缩结果。