Merging common sections of files in BASH

Question

I have been trying to find a way to somehow merge three files at a common point in each of the files and I thought I would ask for some help. My searches have took me on a tour of many possible solutions from sed and awk to arrays. The router.txt file is the skeleton file and I want to read in the vars.txt and the cleanup.txt to produce a single file with the contents of each section together. The merged file should have all sections with all the relevant data in each section from each file.

I have tried arrays with the below:

rtimeserver=(`sed -n "/TIME SERVER/,/! -/p" router.txt | sed '/! -/{2,$d}' `)  
vtimeserver=(`sed -n "/TIME SERVER/,/! -/p" vars.txt | sed '/! -/{2,$d}' `)  
cleantimeserver=(`sed -n "/TIME SERVER/,/! -/p" cleanup.txt | sed '/! -/{2,$d}' `)  

With the below I am able to print just the section of each of the files that I want.

 printf "%s\n" "${rtimeserver[@]}"  
 printf "%s\n" "${vtimeserver[@]}"  
 printf "%s\n" "${cleantimeserver[@]}"  

But I cant seen to be able to figure out how to find the section in the router file and slip in the sections from the other files. I thought I could use SED's search function to find the section and dump the array content but I need help.

I really appreciate any assistance you are willing to provide.
Thanks in advance,
Mike

router.txt

  !  
  ! ---- NAME SERVER ----  

  ! ---- TIME SERVER ----  
  ntp trusted-key  
  ntp authenticate  
  ntp source Loopback0  
  ! ---- TACACS CONFIG ----  
  !  
  ip tacacs source-interface Loopback 0  
  tacacs-server retransmit 2  
  tacacs-server timeout 2  
  no tacacs-server directed-request  

vars.txt

  ! ---- NAME SERVER ----  
  no ip name-server  
  ip domain-name lab.com  
  ip domain name lab.com  
  ip domain-list lab.com  
  ip domain-list int.lab.com  
  ! ---- TIME SERVER ----  
  ntp server 1.1.1.1  
  sntp server 2.2.2.2  
  ! ---- TACACS CONFIG ----  
  tacacs-server key 1111111111  
  tacacs-server host 1.1.1.1  
  tacacs-server host 1.1.1.2  

cleanup.txt

  ! ---- NAME SERVER ----  
  no ip dns server-address 3.3.3.3  
  no ip dns server-address 3.3.3.4  
  no ip dns server-address 3.3.3.5  
  ! ---- TIME SERVER ----  
  no clock summer-time  
  sntp poll-interval 65535  
  clock timezone gmt GMT+00  
  ip sntp source-interface loopback 1  
  ! ---- TACACS CONFIG ----  
  tacacs-server host 3.3.3.3  
  tacacs-server host 3.3.3.4  
  tacacs-server host 3.3.3.5  

merged.txt

  ! ---- TIME SERVER ----  
  ntp trusted-key  
  ntp authenticate  
  ntp source Loopback0  
  ntp server 1.1.1.1  
  sntp server 2.2.2.2  
  no clock summer-time  
  sntp poll-interval 65535  
  clock timezone gmt GMT+00  
  ip sntp source-interface loopback 1  

Answer 1

Hint:

awk '
/! ---/ { p = 0 }        # Disable the flag if lines with header text is seen
p { lines[++l] = $0 }    # If flag is enabled, populate an array with that line
/TIME SERVER/ { header = $0; p = 1 }  # Enable the flag when the line with TIME SERVER is seen and capture the line in header variable
END { 
    print header > "merged.txt"            # Print your header in a new file
    for(line = 1; line <= l; line++)       # Iterate over the array
        print lines[line] >"merged.txt"    # Printing each line in a new file
}' router.txt vars.txt cleanup.txt

Solution:

Extending the hint given above, you can do the following to capture everything:

awk '
/! -/ { p = 0 }
    p { lines[seen] = lines[seen] RS $0 }
/! -/ { seen = $0 ; p = 1}
END {
    for (line in lines)
        print line, lines[line] > "merged.txt"
}' router.txt vars.txt cleanup.txt

This creates a new file called merged.txt that looks like:

$ cat merged.txt
! ---- NAME SERVER ----

no ip name-server
ip domain-name lab.com
ip domain name lab.com
ip domain-list lab.com
ip domain-list int.lab.com
no ip dns server-address 3.3.3.3
no ip dns server-address 3.3.3.4
no ip dns server-address 3.3.3.5
! ---- TIME SERVER ----
ntp trusted-key
ntp authenticate
ntp source Loopback0
ntp server 1.1.1.1
sntp server 2.2.2.2
no clock summer-time
sntp poll-interval 65535
clock timezone gmt GMT+00
ip sntp source-interface loopback 1
! ---- TACACS CONFIG ----
!
ip tacacs source-interface Loopback 0
tacacs-server retransmit 2
tacacs-server timeout 2
no tacacs-server directed-request
tacacs-server key 1111111111
tacacs-server host 1.1.1.1
tacacs-server host 1.1.1.2
tacacs-server host 3.3.3.3
tacacs-server host 3.3.3.4
tacacs-server host 3.3.3.5

Update:

To preserve the order of first file, you can do:

awk '
/! -/ { p = 0 }
    p { lines[seen] = lines[seen] RS $0 }
/! -/ { !($0 in head) ? header[++i]=$0 : x ; p = 1; head[$0]++; seen = $0}
END {
    for (a=1; a<=i; a++)
        print a, header[a], lines[header[a]]
}' router.txt vars.txt cleanup.txt

Answer 2

Using awk

awk 'BEGIN{print "! ---- TIME SERVER ---- "}
/TIME SERVER/{for (i=2;i<NF;i++) print $i}' FS="\n" RS="! ----" router.txt vars.txt cleanup.txt

! ---- TIME SERVER ---- 
  ntp trusted-key  
  ntp authenticate  
  ntp source Loopback0  
  ntp server 1.1.1.1  
  sntp server 2.2.2.2  
  no clock summer-time  
  sntp poll-interval 65535  
  clock timezone gmt GMT+00  
  ip sntp source-interface loopback 1

Explanation

• FS="\\n" RS="! ----" These are key points in this awk command , define RS and FS properly to simplify the process.