我们可以使用AWK和gsub（）处理带有多个冒号“：”的数据吗？ 怎么样？

Question

Here is an example of the data:

Col_01:14 .... Col_20:25    Col_21:23432    Col_22:639142
Col_01:8  .... Col_20:25    Col_22:25134    Col_23:243344
Col_01:17 .... Col_21:75    Col_23:79876    Col_25:634534    Col_22:5    Col_24:73453
Col_01:19 .... Col_20:25    Col_21:32425    Col_23:989423
Col_01:12 .... Col_20:25    Col_21:23424    Col_22:342421    Col_23:7    Col_24:13424    Col_25:67
Col_01:3  .... Col_20:95    Col_21:32121    Col_25:111231

As you can see, some of these columns are not in the correct order...

Now, I think the correct way to import this file into a dataframe is to preprocess the data such that you can output a dataframe with NaN values, eg

Col_01 .... Col_20    Col_21    Col22    Col23    Col24    Col25
8      .... 25        NaN       25134    243344   NaN      NaN
17     .... NaN       75        2        79876    73453    634534
19     .... 25        32425     NaN      989423   NaN      NaN
12     .... 25        23424     342421   7        13424    67
3      .... 95        32121     NaN      NaN      NaN      111231

The solution was shown by @JamesBrown here : How to preprocess and load a "big data" tsv file into a python dataframe?

Using said awk script:

BEGIN {
    PROCINFO["sorted_in"]="@ind_str_asc" # traversal order for for(i in a)                  
}
NR==1 {       # the header cols is in the beginning of data file
              # FORGET THIS: header cols from another file replace NR==1 with NR==FNR and see * below
    split($0,a," ")                  # mkheader a[1]=first_col ...
    for(i in a) {                    # replace with a[first_col]="" ...
        a[a[i]]
        printf "%6s%s", a[i], OFS    # output the header
        delete a[i]                  # remove a[1], a[2], ...
    }
    # next                           # FORGET THIS * next here if cols from another file UNTESTED
}
{
    gsub(/: /,"=")                   # replace key-value separator ": " with "="
    split($0,b,FS)                   # split record from ","
    for(i in b) {
        split(b[i],c,"=")            # split key=value to c[1]=key, c[2]=value
        b[c[1]]=c[2]                 # b[key]=value
    }
    for(i in a)                      # go thru headers in a[] and printf from b[]
        printf "%6s%s", (i in b?b[i]:"NaN"), OFS; print ""
}

And put the headers into a text file cols.txt

Col_01 Col_20 Col_21 Col_22 Col_23 Col_25

My question now: how do we use awk if we have data that is not column: value but column: value1: value2: value3 ?

We would want the database entry to be value1: value2: value3

Here's the new data:

Col_01:14:a:47 .... Col_20:25:i:z    Col_21:23432:6:b    Col_22:639142:4:x
Col_01:8:z .... Col_20:25:i:4    Col_22:25134:u:0    Col_23:243344:5:6
Col_01:17:7:z .... Col_21:75:u:q    Col_23:79876:u:0    Col_25:634534:8:1   

We still provide the columns beforehand with cols.txt

How can we create a similar database structure? Is it possible to use gsub() to limit to the first value before : which is the same as the header?

EDIT: This doesn't have to be awk based. Any language will do naturally

Answer 1

Here is another alternative...

$ awk -v OFS='\t' '{for(i=1;i<NF;i+=2)                  # iterate over name: value pairs
                     {c=$i;                             # copy name in c to modify
                      sub(/:/,"",c);                    # remove colon
                      a[NR,c]=$(i+1);                   # collect data by row number, name
                      cols[c]}}                         # save name
                END{n=asorti(cols,icols);               # sort names
                    for(j=1;j<=n;j++) printf "%s", icols[j] OFS;   # print header 
                    print ""; 
                    for(i=1;i<=NR;i++)                  # print data
                      {for(j=1;j<=n;j++) 
                         {v=a[i,icols[j]];             
                          printf "%s", (v?v:"NaN") OFS} # replace missing data with NaN
                       print ""}}' file | column -t     # pipe to column for pretty print

Col_01   Col_20  Col_21     Col_22      Col_23      Col_25
14:a:47  25:i:z  23432:6:b  639142:4:x  NaN         NaN
8:z      25:i:4  NaN        25134:u:0   243344:5:6  NaN
17:7:z   NaN     75:u:q     NaN         79876:u:0   634534:8:1

Answer 2

I had karakfa's answer as well. If the column name is not separated by whitespace from the value (eg if you have Col_01:14:a:47 ) then you can do this (using GNU awk for the extended match function)

  {
      for (i=1; i<=NF; i++) {
          match($i, /^([^:]+):(.*)/, m)
          a[NR,m[1]] = m[2]
          cols[m[1]]
     }
  }

The END block is the same

Answer 3

Using TXR 's Lisp macro implementation of the Awk paradigm:

(awk (:set ft #/-?\d+/)  ;; ft is "field tokenize" (no counterpart in Awk)
     (:let (tab (hash :equal-based)) (max-col 1) (width 8))
     ((ff (mapcar toint) (tuples 2))  ;; filter fields to int and shore up into pairs
      (set max-col (max max-col [find-max [mapcar first f]]))
      (mapdo (ado set [tab ^(,nr ,@1)] @2) f)) ;; stuff data into table
     (:end (let ((headings (mapcar (opip (format nil "Col~,02a")
                                         `@{@1 width}`)
                                   (range 1 max-col))))
             (put-line `@{headings " "}`))
           (each ((row (range 1 nr)))
             (let ((cols (mapcar (opip (or [tab ^(,row ,@1)] "NaN")
                                       `@{@1 width}`)
                                 (range 1 max-col))))
               (put-line `@{cols " "}`)))))

Smaller sample data:

Col_01: 14  Col_04: 25    Col_06: 23432    Col_07: 639142
Col_02: 8   Col_03: 25    Col_05: 25134    Col_06: 243344
Col_01: 17
Col_06: 19  Col_07: 32425

Run:

$ txr reformat.tl data-small
Col01    Col02    Col03    Col04    Col05    Col06    Col07
14       NaN      NaN      25       NaN      23432    639142
NaN      8        25       NaN      25134    243344   NaN
17       NaN      NaN      NaN      NaN      NaN      NaN
NaN      NaN      NaN      NaN      NaN      19       32425

---

PS opip is a macro which boostraps from the op macro for partial function applications; opip implicitly distributes op into its argument expressions, and then chains the resulting functions together into a functional pipeline: hence " op -pipe". In each pipeline element, its own numbered implicit arguments can be referenced: @1 , @2 , ... if they are absent, then the partially applied function implicitly receives the piped object as its rightmost argument.

The ^(,row ,@1) syntax is TXR Lisp's backquote. The backtick that mainstream Lisp dialects use for backquote is already employed for string quasiquotes. This is equivalent to (list row @1) : make a list consisting of the value of row and of the implicit, op/do -generated function argument @1 . Lists of two elements are being used as the hash keys, which simulates a 2D array. For that, the hash must be :equal-based . The lists (1 2) (1 2) are not eql if they are separate instances rather than one and the same object; they compare equal under the equal function.

Answer 4

Just for fun, some incomprehensible perl

perl -aE'%l=%{{@F}};while(($k,$v)=each%l){$c{$k}=1;$a[$.]{$k}=$v}END{$,="\t";say@c=sort keys%c;for$i(1..$.){say map{$a[$i]{$_}//"NaN"}@c}}' input

(community wiki to hide my shame ...)

Golfed a few chars:

perl -aE'while(@F){$c{$k=shift@F}=1;$data[$.]{$k}=shift@F}END{$,="\t";say@c=sort keys%c;for$i(1..$.){say map{$data[$i]{$_}//"NaN"}@c}}' input