How to replace the list value of a key in yaml file from another file?

Question

I have a file config.yaml that looks like this:

myenv1:
  abc: efg
  host: ( ENV["HOST1"] )
  db: ( ENV["DB1"] )  
  user: <%= ENV["USER1"] %>
  pass: <%= ENV["VE_PASS1"] %>
  time: 00

myenv2:
  abc: efg
  host: ( ENV["HOST2"] )
  db: ( ENV["DB2"] )  
  user: <%= ENV["USER2"] %>
  pass: <%= ENV["VE_PASS2"] %>
  time: 800 

And another file values.yaml that looks like this:

secrets: 
  - MY_PASS_1
  - MY_PASS_2
abc:
  - pqr
psp:
  - jdfhs

I need to update/replace the value (it is a list) of a particular key secret in the file values.yaml with a list of words like VE_PASS1 and VE_PASS2 present in the file config.yaml .

After running the commands, the values.yaml should look like this:

secrets: 
  - VE_PASS1
  - VE_PASS2
abc:
  - pqr
psp:
  - jdfhs

What would be the Linux command or script to do that?

Answer 1

yq can take a list/array of elements as input and can replace the value of key("secrets" in your case) by the input list.

Example:

values.yml

secrets: 
  - MY_PASS_1
  - MY_PASS_2
abc:
  - pqr
psp:
  - jdfhs

# yq -i ".secrets = ["foo","bar"]" yqtest.yaml

After executing the above command the values.yml will be changed to:

secrets: 
  - foo
  - bar
abc:
  - pqr
psp:
  - jdfhs

Now you can declare an empty array in bash script: insert_array=() and can use the same combination of awk and sed (grep -E '\bVE_' $source_file | awk -v RS=[ -v FS=] 'NR>1{print $1}' | sed 's/"//g') within a for loop to insert elements to the array insert_array .

#!/bin/bash
    
insert_array=()
for i in $(grep -E '\bKV_' $source_file | awk -v RS=[ -v FS=] 'NR>1{print $1}' | sed 's/"//g')
  do 
    insert_array+=(\"$i\"); insert_array+=(",") 
  done

Note that in bash, the array doesn't separate the elements with ","(comma) by default as, like the list structure in python, it separates the elements only using a single space character, but yq command doesn't support it.

Example: list without comma

This is how the output of the array looks:

#echo ${insert_array[*]}
foo , bar

list without ","(comma)

# yq -i ".secret = [${insert_array[*]}]" destination.yaml 
Error: bad expression, please check expression syntax

list without ","(comma)

# yq -i ".secret = ["foo" "bar"]" destination.yaml 
    Error: bad expression, please check expression syntax

So I have inserted an "," as an element here after each element insertion in for loop and deleted the last "," element using unset insert_array[-1]

for i in $(grep -E '\bKV_' $source_file | awk -v RS=[ -v FS=] 'NR>1{print $1}' | sed 's/"//g')
 do 
   insert_array+=(\"$i\")
   insert_array+=(",") 
 done

Final Script:

#!/bin/bash

insert_array=()
for i in $(grep -E '\bKV_' $source_file | awk -v RS=[ -v FS=] 'NR>1{print $1}' | sed 's/"//g')
  do 
    insert_array+=(\"$i\")
    insert_array+=(",")
  done
unset insert_array[-1] 
yq -i ".secret = [${insert_array[*]}]" destination.yaml

Answer 2

You can first get the list of filtered words into a temporary file temp like this: - VE_PASS1 - VE_PASS2

And then apply this list to the value of key secrets in values.yaml using yq .

The script/commands would look like this:

#!/bin/bash

source_file=config.yaml
destination_file=values.yaml
temp_file=temp

readarray -t my_array < <(grep -E '\bVE_' $source_file | awk -v RS=[ -v FS=] 'NR>1{print $1}' | sed 's/"//g')

# yq version 4
touch $temp_file
for i in "${my_array[@]}"; do
  printf '\055 %s\n' $i >> $temp_file
done

yq eval '.secrets |= load("temp")' $destination_file --inplace

After running the script, values.yaml will look like this:

secrets: 
  - VE_PASS1
  - VE_PASS2
abc:
  - pqr
psp:
  - jdfhs

Answer 3

You can translate your readarray + awk + sed approach to mikefarah/yq using sub as follows:

yq -i '
  .secrets = (
    load("config.yaml")
    | map(.pass | sub("^.*\[\"(.*)\"\].*$"; "$1"))
  )
' values.yaml

Then, values.yaml would contain:

secrets:
  - VE_PASS1
  - VE_PASS2
abc:
  - pqr
psp:
  - jdfhs