使用ruamel.yaml更新包含多个Yaml的Yaml文件中的Yaml块

Question

I've got a yaml file which contains several yaml blocks. The file can be seen here .

Basically, I am trying to update the value for one of the image keys ( ctrl+f blockfreight/go-bftx: ) in the last yaml block of the file. However, I want to preserve everything about the file (including comments) except for the one value that I am updating.

I've got the following code:

"""
Retrieves a truncated version of the latest git commit sha and updates 
the go-bftx container image tag in app.yaml
"""

import sys
import ruamel.yaml
from subprocess import check_output

yaml_path = 'app.yaml'
for yaml_block in ruamel.yaml.round_trip_load_all(stream=open(yaml_path)):
    pass

# parse the most recent git commit sha from command line
docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()

# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
yaml_block['spec']['template']['spec']['containers'][1]['image'] = docker_image

ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)

This successfully edits the value, keeps comments intact, but only sends the final yaml block to stdout.

Is there any way for me to edit only the sixth (and final) yaml block in app.yaml, and keep the rest of the file intact (comments included)?

I tried replacing pass in the above code with an if statement that sent the first five yamls to stdout, and then edited the value in the sixth value, sending that to stdout as well. My thinking was to use bash to send all of the stdout to a file (eg python app_modifier.py > app1.yaml ), but this only sent the output from the sixth yaml.

That code looked like this:

for i, yaml_block in enumerate(ruamel.yaml.round_trip_load_all(stream=open(yaml_path))):
    if i != 5:
        ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)
    else:
        # parse the most recent git commit sha from command line
        docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 --pretty=format:%h'.split()).decode()

        # update go-bftx image with most recent git-commit-sha tag in the StatefulSet blocks
        yaml_block['spec']['template']['spec']['containers'][1]['image'] = docker_image

        ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)

Any assistance would be greatly appreciated! Thank you!

Answer 1

Your file consists of multiple YAML documents, this is what you read in using round_trip_load_all , which gives you a generator.

If you write that back using round_trip_dump() , you are never going to get the --- YAML document separators that are in the original file.

You can probably use a generator and round_trip_dump_all since you know in which document to make changes, but it is probably easier to make a list of what round_trip_load_all generates and work on that. I would also use ruamel.yaml 's new API to do so:

import sys
import ruamel.yaml

yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = list(yaml.load_all(open('app.yaml')))

# parse the most recent git commit sha from command line
# docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
docker_image = 'blockfreight/go-bftx:ci-cd-' + 'check_output_output'

# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
data[-1]['spec']['template']['spec']['containers'][1]['image'] = docker_image

with open('out.yaml', 'w') as ofp:
    yaml.dump_all(data, ofp)

The above mostly works. The out.yaml diff's with the change you want to make, and some whitespace standardisations (indents, between [ " ) which are of course one off.

There are two major issues with this approach:

• The first YAML document in the file has no content, so its comments cannot be hung from any mapping/sequence and are not preserved
• The second YAML document has a final comment ( # // Initializes BFTX Service to interact with endpoints ) that gets dropped. That is most likely a bug in ruamel.yaml that shows up in multi-document files only and that I have to investigate

To solve the first issue, you can only read that document as normal lines without ruamel.yaml . When you do so, you might as well do all but the rest, as this solves (ie circumvents) the second issue as well:

import sys
import ruamel.yaml

with open('out.yaml', 'w') as ofp:
    lines = ''
    with open('app.yaml') as ifp:
        for line in ifp:
            lines += line
            if line == '---\n':
                ofp.write(lines)
                lines = ''
    # process lines from the last document
    # print(lines)
    yaml = ruamel.yaml.YAML()
    yaml.preserve_quotes = True
    data = yaml.load(lines)

    # parse the most recent git commit sha from command line
    # docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
    docker_image = 'blockfreight/go-bftx:ci-cd-' + 'check_output_output'

    # update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
    data['spec']['template']['spec']['containers'][1]['image'] = docker_image

    yaml.dump(data, ofp)

This should do what you want unless you care about the leading trailing space in a flow style sequence, or preserve the three places with inconsistent indentation. The output of diff -u app.yaml out.yaml :

--- app.yaml    2018-06-23 14:41:02.256290577 +0200
+++ out.yaml    2018-06-23 14:58:09.933991459 +0200
@@ -143,7 +143,7 @@
 spec:
   selector:
     matchLabels:
-       app: bftx
+      app: bftx
   serviceName: blockfreight
   replicas: 1
   template:
@@ -151,7 +151,7 @@
       labels:
         app: bftx
     spec:
-     containers:
+      containers:
       - name: tm
         imagePullPolicy: IfNotPresent
         image: tendermint/tendermint:0.20.0
@@ -199,7 +199,7 @@
           tendermint node --moniker="`hostname`" --p2p.seeds="aeabbf6b891435013f2a800fa9e22a1451ca90fd@bftx0.blockfreight.net:8888,6e9515c2cfed19464e6ce11ba2297ecdb411103b@bftx1.blockfreight.net:8888,b8b988370783bd0e58bf926d621a47160af2bdae@bftx2.blockfreight.net:8888,8c091f4e3dc4ac27db1efd38beee012d99967fd8@bftx3.blockfreight.net:8888" --proxy_app="tcp://localhost:46658" --consensus.create_empty_blocks=false
       - name: app
         imagePullPolicy: Always
-        image: blockfreight/go-bftx:rc1
+        image: blockfreight/go-bftx:ci-cd-check_output_output
         ports:
         - containerPort: 12345
         - containerPort: 46658
@@ -247,7 +247,7 @@
         - mountPath: /etc/nginx/conf.d/pub_key.conf
           name: tmconfigdir
           subPath: pub_key_nginx.conf
-     volumes:
+      volumes:
       - name: tmconfigdir
         configMap:
           name: bftx-config
@@ -262,7 +262,7 @@
       annotations:
         volume.alpha.kubernetes.io/storage-class: anything
     spec:
-      accessModes: [ "ReadWriteOnce" ]
+      accessModes: ["ReadWriteOnce"]
       resources:
         requests:
           storage: 2Gi
@@ -271,7 +271,7 @@
       annotations:
         volume.alpha.kubernetes.io/storage-class: anything
     spec:
-      accessModes: [ "ReadWriteOnce" ]
+      accessModes: ["ReadWriteOnce"]
       resources:
         requests:
           storage: 12Mi