yaml multiple complex map keys as a struct

Question

I want to unmarshal complex map key into struct:

YAML

unf:
    item_meta:
      source:
        ? id: "aa"
          name: "bb"
        : "some value"

Struct

type Source struct {
    ID     string `yaml:"id"`
    Name   string `yaml:"name"`
}

Everything works as expected until I add another key:

YAML 2

unf:
    item_meta:
      source:
        ? id: "012"
          name: "Bill"
        : "some value"
        ? id: "066"
          name: "Bob"
        : "another value"

I got an error

"line xxx: mapping key "" already defined at line xxx"

I decided to use aliases:

YAML 3

unf:
    item_meta:
      aliases:
        - bill: &alias_bill
          id: "012"
          name: "Bill"
        - bob: &alias_bob
          id: "066"
          name: "Bob"
      source:
        ? *alias_bill
        : "some value"
        ? *alias_bob
          name: "Bob"
        : "another value"

It fixed the problem! BUT we use hiera server in our stack AND hiera returns contents of a config file already substituted so I end up with YAML 2 version.

Any ideas on how to fix the problem? To config the hiera server is not an option.

Go playground

Answer 1

My solution mostly was based on this issue @larsks

The idea was to find nodes with duplicate map keys and to create a custom value from value nodes of THE map node.

func fixYamlNode(node *yaml.Node) {
    if node.Kind == yaml.MappingNode {
        length := len(node.Content)

        for i := 0; i < length; i += 2 {
            nodes := make([]*yaml.Node, 0)
            nodes = append(nodes, node.Content[i])
            for j := i + 2; j < length; j += 2 {
                nj := node.Content[j]
                if nodes[0].Kind == nj.Kind && nodes[0].Value == nj.Value {
                    nodes = append(nodes, nj)
                }
            }
            if len(nodes) == 1 {
                continue
            }

            fillMapValue(nodes)
        }

        for i := 1; i < length; i += 2 {
            valueNode := node.Content[i]
            fixYamlNode(valueNode)
        }
    }
}

func fillMapValue(nodes []*yaml.Node) {
    for _, node := range nodes {
        length := len(node.Content)
        for i := 0; i < length; i += 2 {
            node.Value = fmt.Sprintf("%s %s", node.Value, node.Content[i+1].Value)
        }
    }
}

Answer 2

This is an error in go-yaml. See these lines :

            ni := n.Content[i]
            for j := i + 2; j < l; j += 2 {
                nj := n.Content[j]
                if ni.Kind == nj.Kind && ni.Value == nj.Value {
                    d.terrors = append(d.terrors, fmt.Sprintf("line %d: mapping key %#v already defined at line %d", nj.Line, nj.Value, ni.Line))
                }
            }

This code checks for duplicate keys. For each key in the mapping ( ni ), it goes through all succeeding keys ( nj ) and checks whether they have the same Kind and the same Value . This is wrong, since Value is only set for scalar keys, so as soon as you have two keys that are mappings, this error fires even though the keys are different, because the code does not compare the key's content, only their Values (which will be the empty strings for non-scalar nodes).

I suggest you open an issue with go-yaml.