Process multiple files and append them in linux/unix

Question

I have over 100 files with at least 5-8 columns (tab-separated) in each file. I need to extract first three columns from each file and add fourth column with some predefined text and append them.

Let's say I have 3 files: file001.txt , file002.txt , file003.txt .

file001.txt :

chr1 1 2 15
chr2 3 4 17

file002.txt :

chr1 1 2 15
chr2 3 4 17

file003.txt :

chr1 1 2 15
chr2 3 4 17

combined_file.txt :

chr1 1 2 f1
chr2 3 4 f1
chr1 1 2 f2
chr2 3 4 f2
chr1 1 2 f3
chr2 3 4 f3

For simplicity I kept file contents same. My script is as follows:

#!/bin/bash
for i in {1..3}; do
j=$(printf '%03d' $i)
awk 'BEGIN { OFS="\t"}; {print $1,$2,$3}' file${j}.txt | awk -v k="$j" 'BEGIN {print $0"\t$k”}' | cat >> combined_file.txt
done

But the script is giving the following errors:

awk: non-terminated string $k”}... at source line 1 context is

<<< awk: giving up source line number 2 awk: non-terminated string $k”}... at source line 1 context is <<< awk: giving up source line number 2

Can some one help me to figure it out?

Answer 1

You don't need two different awk scripts. And you don't use $ to refer to variables in awk , that's used to refer to input fields (ie $k means access the field whose number is in the variable k ).

for i in {1..3}; do
    j=$(printf '%03d' $i)
    awk -v k="$j" -v OFS='\t' '{print $1, $2, $3, k}' file$j.txt
done > combined_file.txt

Answer 2

As pointed out in the comments your problem is youre trying to use odd characters as if they were double quotes. Once you fix that though, you don't need a loop or any of that other complexity all you need is:

$ awk 'BEGIN{FS=OFS="\t"} {$NF="f"ARGIND} 1' file*
chr1    1       2       f1
chr2    3       4       f1
chr1    1       2       f2
chr2    3       4       f2
chr1    1       2       f3
chr2    3       4       f3

The above used GNU awk for ARGIND.