How to edit a YAML file in python repeatedly using a variable without defining a specific value for the key?

Question

This is my example YAML file,

# Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1
Martin:
  name: Martin Devloper
  job: Developer
  skills:
    - python

I want to read this yaml file from a particular directory, edit it and then save the edited text on the same yaml file. I got this code from a link somewhere as follows:

import sys
from ruamel.yaml import YAML
inp_fo = open("C:/<Users>/Master 
1TB/source/repos/New_insertdata/PythonApplication3/inp.yaml").read() 
#Read the YAML File

yaml = YAML()
code = yaml.load(inp_fo)
sys.stdout.write('\n')
code.insert(1, 'sensor', 'None', comment="new key")
inp_fo2 = open("inp.yaml","w") #Open the file for Write
yaml.dump(code, inp_fo2) ##Write to file with new parameter
inp_fo2.close() ## close the file

When I execute it I get output as follows:

# Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1
Martin:
  name: Martin 
  job: Developer
  skills:
    - python
Tabitha: None 

But instead I need:

# Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1
Martin:
  name: Martin 
  job: Developer
  skills:
    - python
Tabitha:
  name: Tabitha Bitumen
  job: Developer
  skills:
    - Java

How to code this?

I entered value for 'Tabitha' as None because code.insert was not getting executed if I don't insert a value for it.

Now, I want to edit this yaml file repeatedly so I want to define a variable for each key value, so everytime once the variable value changes the code will be executed and new key values will be inserted in the existing yaml file. I don't want to delete the earlier contents of YAML file just need to go on adding new contents (key & value) (content will be employee details like name, job, skills).

I am completely new to this coding language so if I am wrong please correct me. Also can anyone explain me what is the use of indent, sequence & mapping in ruamel.yaml and how can I use them in code?

I am a learner and the above example is a random yaml file just an example to learn ruamel.yaml.

Let me explain you what I want exactly, The following content will remain static.

 # Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1

But the key values (employee details) are going to change. Example I want: Martin: name: Martin Devloper job: Developer skills: - python

"Martin" should be a "variableP"
name: "variableQ"
job: "variableR"
skills:
  - "variableS"

similarly if another employee adds his details the code will automatically add another employee details in those variables. because according to your code, I will have to repeatedly add

code['Tabitha'] = dict(name='Tabitha Bitumen', job='Developer', skills=['Java'])

again and again for other employee data which I don't want, I just want a code where I could define a variable whose value will keep on changing and will edit the same yaml file repeatedly and do not delete the earlier employee details. There is no limit for list of employees to be get added so I don't want to edit the code again and again.

I basically know C and arduino programming language so looking for a way to define variables in a code which will make my work easy.

Answer 1

The YAML library you're using converts between YAML formatting and Python dictionaries. Therefore you could simple edit the dictionary to add or change a value in the YAML file. For example:

import sys
from ruamel.yaml import YAML
filename = "C:/<Users>/Master/1TB/source/repos/New_insertdata/PythonApplication3/inp.yaml"

yaml = YAML()
with open(filename) as inp_fo:
    code = yaml.load(inp_fo)

    code["Tabitha"] = {
                        "name":"Tabitha Bitman",
                        "job":"Developer",
                        "skills":["Python"]
                      }

with open(filename, "w") as file:
    yaml.dump(code, file) ##Write to file with new parameter

There is no need to close the files as they were opened in "with" statements

This will update Tabitha's details in the YAML file, assuming that the rest of your code is correct.

You can also parse a variable into the key:value lookup, replacing code["Tabitha"] with code[variable_name] .

If you are working with YAML (or JSON) files I also highly recommend looking into and understanding dictionaries in Python.

Answer 2

The following assumes you are using Python3 (and you should if you are learning Python), because it uses pathlib that is in the standard library for Python3. There is a pathlib2 package that provides that functionality for Python2.

The example code that you present does a few strange things:

• it is not necessary to read in the content of a file ( open(...).read() ) and then hand that to the .load() method, you can directly pass in the file pointer (ie the result from just doing open() , or you can use the Path object and hand this to .load() without even opening the file (as shown below).

• it is unclear why a newline is written to sys.stdout , that really has no function in this snippet

• the call code.insert(1, 'sensor', 'None', comment="new key") does put 'sensor' as a new key, in position 1 of the dictionary like object code , with the value None and the end-of-line comment "new key". (Position counting starts at 0.

If you had really ran that code you would get as output:

# Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
sensor: None  # new key
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1
Martin:
  name: Martin Devloper
  job: Developer
  skills:
  - python

At the root of your input file is a mapping with keys Developer , Support and Martin . The values associate with a key can again be a mapping (as is the case for all your root-level keys), but they can also be a sequence (as is the value for skills , indicated by dash ( - ) or scalar values (string, number, etc). The root of a YAML document can be a scalar (which is then the only object in the document), or a sequence. And a sequence has elements that can be scalars (as your final line python ), but those elements can also be mappings or other sequences.

Mappings are loaded as Python dict s, sequences are loaded as Python list s. In order to preserve comments (and other information), ruamel.yaml will create dict- resp. list-like objects when used as in the example. You would not be able to call code.insert() on a regular Python dict but you can do so on what you got back (from YAML().load() )

If you just want to add a key-value pair at the end of a dict, you don't need to use .insert , you can just do:

code['Tabitha'] = dict(name='Tabitha Bitumen', job='Developer', skills=['Java'])

If you need to move something to the end of the YAML file, you actually cannot just update the value using eg

code['Support'] = dict(Department='General', Manager='KLM', Floor='1st', Lab=1)

this will get you the key/value in the old position. You would have to do

del code['Support']

first to make it forget this old position.

---

As indicated ruamel.yaml supports multiple ways on how to read and write to a file. The with statement as Moralous answer introduced is already better than explicit opening, reading and closing. You can also use pathlib.Path like objects with the .load() and .dump() methods:

from pathlib import Path
from ruamel.yaml import YAML

path = Path('inp.yaml')  # I shortened this a bit, as I don't have a C: drive
opath = Path('outp.yaml')

yaml = YAML()
code = yaml.load(path)
code['Tabitha'] = dict(name='Tabitha Bitumen', job='Developer', skills=['Java'])
yaml.dump(code, opath)

which gives:

# Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1
Martin:
  name: Martin Devloper
  job: Developer
  skills:
  - python
Tabitha:
  name: Tabitha Bitumen
  job: Developer
  skills:
  - Java

This is almost what you want, but the default indentation of the YAML output is 2 positions for both the dict (as you want it to be) and for the sequence (it is counted to the beginning of your element, not to the dash).

To get what you want, you need to specify an indent of four for the sequence, and an offset of two for the dash withing those for spaces. You can do so by inserting, after the yaml = YAML() line, a line with:

yaml.indent(sequence=4, offset=2)

---

You can also use your YAML instance itself in a with statement:

import sys
from pathlib import Path
from ruamel.yaml import YAML

path = Path('inp.yaml')  # I shortened this a bit, as I don't have a C: drive
opath = Path('outp.yaml')

with YAML(output=opath) as yaml:
    yaml.indent(sequence=4, offset=2)
    code = yaml.load(path)
    code['Tabitha'] = dict(name='Tabitha Bitumen', job='Developer', skills=['Java'])
    yaml.dump(code)

That will also give you exactly you want:

# Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1
Martin:
  name: Martin Devloper
  job: Developer
  skills:
    - python
Tabitha:
  name: Tabitha Bitumen
  job: Developer
  skills:
    - Java

You tagged your question with PyYAML as well. Please note that with that package you cannot do the above: your comments in the YAML file will be lost, and you don't have the kind of fine control over indentation to get the result you wanted.

Answer 3

While working on this YAML i found the question and thought to share the explanation if this could helps other in understanding in a better way. YAML are the configuration files and we can parse JSON format using a YAML parser in python. We can use ruamel.yaml PYPI module for building the parser in python.

pip install ruamel.yaml

Here is the sample YAML config.yaml file:-

# Employee records
Developer:
  Department: IT
  Manager: ABCD
  Floor: 2nd
  Lab: 4
Support:
  Department: General
  Manager: XYZ
  Floor: 1st
  Lab: 1
Martin:
  name: Martin Devloper
  job: Developer
  skills:
    - python
Kavya:
  name: Kavya Agarwal
  job: Developer
  skills:
    - Java
Atharv:
  Name: 190.22.45.33
  Job: Smriti
  Skills:
    - python

Here i am attaching the YAML parser code:-

from ruamel.yaml import YAML
filepath = 'config.yaml'
def yaml_loader(file_path):
    yaml = YAML()
    with open(file_path) as fp:
       yaml.indent(sequence=4, offset=2)
       data = yaml.load(fp)
       data['Atharv'] = {
                       "Name":"190.22.45.33",
                       "Job":"Smriti",
                       "Skills":["python"]
                     }
    with open(file_path, "w") as file:
       yaml.dump(data, file)

yaml_loader("config.yaml")

---