how to insert data in between already existing data present in .yaml file through python?

Question

I'm trying to create an logging software using Python and YAML. I need to test multiple boards and I might need to take more than 1 trial per board. So I want to arrange the data in YAML file in ascending orders of board numbers to automate the analysis process.

#!/usr/bin/python3

import yaml
import io
board_no = (input('Enter the board number: '))
trial_no = (input('Enter the trial no: '))
reading1 = int(input('Reading1: '))
status = input('Pass/Fail: ')
yaml.allow_duplicate_keys = True
basic_info = { 'Board ' + board_no:{
        'Trial ' + trial_no:{
            'Reading1 '  :  reading1,
            'Status ': status
           }
        }
   }
fp = "../yaml_101/test_list.yaml"
outfile = open (fp,'a')
yaml.dump(basic_info, outfile, indent = 4, default_flow_style = False)
outfile.close()

I've written this rather simple code just to make log entries. I've looked for ways to insert the data in between 2 blocks but didn't find one. My current output is (after manual entries):

Board 1:
    Trial 1:
        'Reading1 ': 450
        'Status ': Pass
Board 2:
    Trial 1:
        'Reading1 ': 758
        'Status ': Fail
Board 3:
    Trial 1:
        'Reading1 ': 450
        'Status ': Pass
Board 2:
    Trial 2:
        'Reading1 ': 450
        'Status ': Pass

And output I want is:

Board 1:
    Trial 1:
        'Reading1 ': 450
        'Status ': Pass
Board 2:
    Trial 1:
        'Reading1 ': 758
        'Status ': Fail
Board 2:
    Trial 2:
        'Reading1 ': 450
        'Status ': Pass
Board 3:
    Trial 1:
        'Reading1 ': 450
        'Status ': Pass

Can someone please guide me? I'm totally new to YAML and Python.

Answer 1

There are several problems with your code:

1. You open the file for appending, but you don't want to append, you want to insert. Of course that is possible, but then you need to know where to position the file pointer, read everything after that in a buffer, write your new stuff, then write the buffer. Knowing where to position your file pointer is going to be the difficult part.

2. You do yaml.allow_duplicate_keys = True but that doesn't do anything in PyYAML. It is a ruamel.yaml feature, that allows you to read files with duplicate keys (disallowed by the YAML specification, but silently ignored by the non-conforming PyYAML library).

3. You should not manipulate the YAML file directly, and always use load the data from YAML, update it and dump. That way your YAML will be conforming, even if, for instance, one of your inputs contains a single or double quote, or spaces (as you can see from 'Trial ' , that needs quotes in YAML because of the space).

4. You are only using mappings in your YAML file, but such lists of trials, lend themselves much better to sequences.

5. Your keys have enumeration information, this is almost always a bad idea, because you cannot know up-front what the key looks like. So instead of a key Board 1 you should have a key Board with a value 1 , or as value a mapping with numbers of the boards as key. That way it is easy to get the data for a particular board number (or update the trials if handled in the same way)

---

To start you should make the status.yaml file look like:

- Board: 1
  Trial: 1
  Reading1: 450
  Status: Pass
- Board: 2
  Trial: 1
  Reading1: 758
  Status: Fail
- Board: 3
  Trial: 1
  Reading1: 450
  Status: Pass

or use:

Boards:
  - nr: 1
    Trials:
    - nr: 1
      Reading1: 450
      Status: Pass
  - nr: 2
    Trials:
    - nr: 1
      Reading1: 758
      Status: Fail
  - nr: 3
    Trials:
    - nr: 1
      Reading1: 450
      Status: Pass

or (my favourite):

Boards:
  1:
    Trials:
      1:
        Reading: 450
        Status: Pass
  2:
    Trials:
      1:
        Reading: 758
        Status: Fail
  3:
    Trials:
      1:
         Reading: 450
         Status: Pass

In the first example you should walk over the Python list en insert a new trial, before the value of Board changes to be greater than your input board number.

To update the last example use ruamel.yaml (disclaimer I am the author of that package), so the keys in your mappings stay in the same order:

from pathlib import Path
import ruamel.yaml

status_file = Path('status.yaml')
yaml = ruamel.yaml.YAML()
# yaml.indent(mapping=2)
yaml.preserve_quotes = True

data = yaml.load(status_file)

board_no = 2
 # no need to input the trial_no, it should be the next number for this board
trials = data['Boards'][board_no]['Trials']
trial_no = sorted(trials.keys())[-1] + 1
reading1 = 450
status = 'Pass'

trials[trial_no] = dict(Reading1=reading1, Status=status)

yaml.dump(data, status_file)

which updates status.yaml to become:

Boards:
  1:
    Trials:
      1:
        Reading1: 450
        Status: Pass
  2:
    Trials:
      1:
        Reading1: 758
        Status: Fail
      2:
        Reading1: 450
        Status: Pass
  3:
    Trials:
      1:
        Reading1: 450
        Status: Pass