How to remove several columns and the field separators at once in AWK?

Question

I have a big file with several thousands of columns. I want to delete some specific columns and the field separators at once with AWK in Bash.

I can delete one column at a time with this oneliner (column 3 will be deleted and its corresponding field separator):

awk -vkf=3 -vFS="\t" -vOFS="\t" '{for(i=kf; i<NF;i++){ $i=$(i+1);}; NF--; print}' < Big_File

However, I want to delete several columns at once... Can someone help me figure this out?

Answer 1

You can pass list of columns to be deleted from shell to awk like this:

awk -vkf="3,5,11" ...

then in the awk programm parse it into array:

split(kf,kf_array,",")

and then go thru all the colums and test if each particular column is in the kf_array and possibly skip it

Other possibility is to call your oneliner several times :-)

Answer 2

Here is an implementation of Kamil's idea:

awk -v remove="3,8,5" '
  BEGIN {
    OFS=FS="\t"
    split(remove,a,",")
    for (i in a) b[a[i]]=1
  }                                                          
  {
    j=1
    for (i=1;i<=NF;++i) {
      if (!(i in b)) { 
        $j=$i
        ++j
      }
    }
    NF=j-1
    print
  }
'

Answer 3

If you can use cut instead of awk , this one is easier with cut :

eg this obtains columns 1,3, and from 50 on from file:

cut -f1,3,50- file

Answer 4

Something like this should work:

awk -F'\t' -v remove='3|8|5' '
{
   rec=ofs=""
   for (i=1;i<=NF;i++) {
      if (i !~ "^(" remove ")$" ) {
         rec = rec ofs $i
         ofs = FS
      }
   }
   print rec
}
' file