Concatenating 500 files while removing the first line from each file except the first file

Question

I want to create a file which is the result of concatenation of 500 files where the first line of each file except the first file is deleted. I also want the original files unchanged.

I know that cat and sed should be piped but I cannot wrap my mind around it!

for the moment what I can think of is as follows:

1. Backup the original files.

2. Remove the header from every file using:

   for x in *.seg; do sed -i 1d ${x}; done

3. concatenate files using cat
4. add the header to the result of step 3.

Can you propose a pipe that can do this while keeping the original files intact?

Answer 1

You could use awk to do this:

awk 'NR == FNR || FNR > 1' *.seg > destination

For the first file, the total record number NR will equal the record number of the current file FNR , so all lines will be printed. For other files, only lines after the first will be printed. The output is redirected to a file destination .

As you have 500 files, the FNR > 1 will evaluate to true more often than the NR == FNR , so you may want to switch around the order so that short-circuiting takes place:

awk 'FNR > 1 || NR == FNR' *.seg > destination

When the first part of the || is true, there is no need to evaluate the second part. Much faster ;)