How to force the comment position to be valid with yamllint when load and dump with ruamel.yaml?

Question

I would like to be able to "reposition" existing comments in a yaml document so that they are valid when running yamllint on the output produced, the default configuration being min-spaces-from-content=2 (ref: https://yamllint.readthedocs.io/en/stable/rules.html#module-yamllint.rules.comments ).

According to How can I add a comment with ruamel.yaml , using something like yaml_add_eol_comment('the comment', 'the key', column=None) should do the trick, but that's not the result I've got so far.

Here is a piece of code (using ruamel.yaml in version 0.16.7 ) I wrote to demonstrate the current behaviour I'm having :

"""Play with comments."""
from __future__ import print_function
import sys
import ruamel.yaml

yaml = ruamel.yaml.YAML()
yaml.indent(mapping=2, sequence=4, offset=2)
inp = """\
---
list-of-maps:
  - part_no:   A4786 # comment 1
    part_henon: mouhaha    # you're not funny
  - part_yes: A21 # also a comment here
    part_iculier: partenaire # I don't always understand how it works
    part_third: key # komment
list-only:
  - first # comment 2
  - third # I have a comment too
  - second # what?
simple-map:
  what-are-you-waiting-for: christmas? # duke nukem rulez
  jingle: bels # not christmas yet
map-of-maps:
  key:
    another-sub-key: w00t # casimir
    sub-key: sub-value # comment 3
    my-sub-key-name-is-longuer-than-yours: 1 # sentinel vs superman
"""

data = yaml.load(inp)


def process_comments(data, column=None):
    if isinstance(data, dict):
        if data.ca:
            if data.ca.items:
                for key in data.ca.items.keys():
                    if data.ca.items[key][2]:
                        comment = data.ca.items[key][2].value.replace("\n", "")
                        data.yaml_add_eol_comment(comment, key, column=column)
        for k, v in data.items():
            process_comments(k, column=column)
            process_comments(v, column=column)
    elif isinstance(data, list):
        if data.ca:
            if data.ca.items:
                for key in data.ca.items.keys():
                    if data.ca.items[key][0]:
                        comment = data.ca.items[key][0].value.replace("\n", "")
                        data.yaml_add_eol_comment(comment, key, column=column)
        for elem in data:
            process_comments(elem, column=column)


process_comments(data, column=None)
yaml.dump(data, sys.stdout)

The expected output is :

list-of-maps:
  - part_no: A4786  # comment 1
    part_henon: mouhaha  # you're not funny
  - part_yes: A21  # also a comment here
    part_iculier: partenaire  # I don't always understand how it works
    part_third: key  # komment
list-only:
  - first  # comment 2
  - third  # I have a comment too
  - second  # what?
simple-map:
  what-are-you-waiting-for: christmas?  # duke nukem rulez
  jingle: bels  # not christmas yet
map-of-maps:
  key:
    another-sub-key: w00t  # casimir
    sub-key: sub-value  # comment 3
    my-sub-key-name-is-longuer-than-yours: 1  # sentinel vs superman

The actual output is :

list-of-maps:
  - part_no: A4786  # comment 1
    part_henon: mouhaha # you're not funny
  - part_yes: A21  # also a comment here
    part_iculier: partenaire # I don't always understand how it works
    part_third: key # komment
list-only:
  - first # comment 2
  - third # I have a comment too
  - second # what?
simple-map:
  what-are-you-waiting-for: christmas?  # duke nukem rulez
  jingle: bels # not christmas yet
map-of-maps:
  key:
    another-sub-key: w00t  # casimir
    sub-key: sub-value # comment 3
    my-sub-key-name-is-longuer-than-yours: 1 # sentinel vs superman

So it seems that :

• for CommentedMaps, the comment for the first key gets two spaces before the #, but not the other comments associated to the other keys of the CommentedMap
• for CommentedSeqs, there is always a single space before the #

Am I missing something?

Additionnal informations :

By the way, a simpler example gives the same output/behaviour :

"""Test comments on a very simple CommentedMap."""
import sys
import ruamel.yaml


comment_column = None
insert = ruamel.yaml.comments.CommentedMap()
insert["test"] = "asdf"
insert.yaml_add_eol_comment("Test Comment!", "test", column=comment_column)
insert["second-key"] = "yop"
insert.yaml_add_eol_comment("Another comment", "second-key", column=comment_column)

yaml = ruamel.yaml.YAML()
yaml.dump(insert, sys.stdout)

outputs :

test: asdf  # Test Comment!
second-key: yop # Another comment

Disclaimers :

• thank you very much to the author of ruamel.yaml , it's an awesome lib ... and thank you for watching after SO questions about the lib
• I am not pretending to be a good python developer at all so please forgive me if my code is not of a good quality

Answer 1

That you get two spaces on the first comment of a mapping is a bug. If you don't specify a column the column is sort of guessed based on the preceding key. For the first key-value pair in a mapping that is not available, and that results in a slightly different code path.

Fixing that bug will not help you, you'll have to provide your own yaml_add_eol_comment that always adds an extra space:

import sys
import ruamel.yaml


yaml = ruamel.yaml.YAML()
yaml.indent(mapping=2, sequence=4, offset=2)
inp = """
list-of-maps:
  - part_no: A4786 # comment 1
    part_henon: mouhaha    # you're not funny
  - part_yes: A21 # also a comment here
    part_iculier: partenaire # I don't always understand how it works
    part_third: key # komment
list-only:
  - first # comment 2
  - third # I have a comment too
  - second # what?
simple-map:
  what-are-you-waiting-for: christmas? # duke nukem rulez
  jingle: bels # not christmas yet
map-of-maps:
  key:
    another-sub-key: w00t # casimir
    sub-key: sub-value # comment 3
    my-sub-key-name-is-longuer-than-yours: 1 # sentinel vs superman
"""

data = yaml.load(inp)


def my_add_eol_comment(self, comment, key=ruamel.yaml.comments.NoComment, column=None):
    org_col = column
    if column is None:
        try:
            column = self._yaml_get_column(key)
        except AttributeError:
            column = 0
    if comment[0] != '#':
        comment = '# ' + comment
    if org_col != 0:  # only do this if the specified colunn is not the beginning of the line
        if comment[0] == '#':
            comment = ' ' + comment
            column = 0
    start_mark = ruamel.yaml.error.CommentMark(column)
    ct = [ruamel.yaml.tokens.CommentToken(comment, start_mark, None), None]
    self._yaml_add_eol_comment(ct, key=key)

ruamel.yaml.comments.CommentedBase.yaml_add_eol_comment = my_add_eol_comment


def process_comments(data, column=None):
    if isinstance(data, dict):
        if data.ca:
            if data.ca.items:
                for key in data.ca.items.keys():
                    if data.ca.items[key][2]:
                        comment = data.ca.items[key][2].value.replace("\n", "")
                        data.yaml_add_eol_comment(comment, key, column=column)
        for k, v in data.items():
            process_comments(k, column=column)
            process_comments(v, column=column)
    elif isinstance(data, list):
        if data.ca:
            if data.ca.items:
                for key in data.ca.items.keys():
                    if data.ca.items[key][0]:
                        comment = data.ca.items[key][0].value.replace("\n", "")
                        data.yaml_add_eol_comment(comment, key, column=column)
        for elem in data:
            process_comments(elem, column=column)

process_comments(data, column=None)
yaml.dump(data, sys.stdout)

which gives:

list-of-maps:
  - part_no: A4786  # comment 1
    part_henon: mouhaha  # you're not funny
  - part_yes: A21  # also a comment here
    part_iculier: partenaire  # I don't always understand how it works
    part_third: key  # komment
list-only:
  - first  # comment 2
  - third  # I have a comment too
  - second  # what?
simple-map:
  what-are-you-waiting-for: christmas?  # duke nukem rulez
  jingle: bels  # not christmas yet
map-of-maps:
  key:
    another-sub-key: w00t  # casimir
    sub-key: sub-value  # comment 3
    my-sub-key-name-is-longuer-than-yours: 1  # sentinel vs superman