Format List Output into Dictionary

Question

I've written a megacli prometheus exporter that requires the following data: firmware state, errors, enclosure id, slot, and adapter.

The output of the script looks something like this:

['Adapter #0', 'Enclosure Device ID: 252', 'Slot Number: 0', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up', 'Enclosure Device ID: 252', 'Slot Number: 1', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up',
'Adapter #1', 'Enclosure Device ID: 252', 'Slot Number: 0', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up', 'Enclosure Device ID: 252', 'Slot Number: 1', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up',
'Adapter #2', 'Enclosure Device ID: 252', 'Slot Number: 0', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up', 'Enclosure Device ID: 252', 'Slot Number: 1', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up']

To turn this into something that is query-able by prometheus, I am trying to reformat the above list into something like this:

{
    "Adapter #0": {
        1: {
            "Enclosure Device ID": "252",
            "Slot Number": "0",
            "Media Error Count": "0",
            "Other Error Count": "0",
            "Firmware State": "Online, Spun Up"
        },
        2: {
            "Enclosure Device ID": "252",
            "Slot Number": "1",
            "Media Error Count": "0",
            "Other Error Count": "0",
            "Firmware State": "Online, Spun Up"
        }
    }
},

{
    "Adapter #1": {
        1: {
            "Enclosure Device ID": "252",
            "Slot Number": "0",
            "Media Error Count": "0",
            "Other Error Count": "0",
            "Firmware State": "Online, Spun Up"
        },
        2: {
            "Enclosure Device ID": "252",
            "Slot Number": "1",
            "Media Error Count": "0",
            "Other Error Count": "0",
            "Firmware State": "Online, Spun Up"
        }
    }
},

{
    "Adapter #2": {
        1: {
            "Enclosure Device ID": "252",
            "Slot Number": "0",
            "Media Error Count": "0",
            "Other Error Count": "0",
            "Firmware State": "Online, Spun Up"
        },
        2: {
            "Enclosure Device ID": "252",
            "Slot Number": "1",
            "Media Error Count": "0",
            "Other Error Count": "0",
            "Firmware State": "Online, Spun Up"
        }
    }
}
   

Ultimately, the final query will turn into the below for each drive

"megacli_query"{'enclosure_id'='Enclosure Device ID','adapter_#'='adapter','slot_number'='Slot Number', 'media_error'='Media Error Count','other_error'='Other Error Count','state'='Firmware State'}

So there are a couple steps I'm trying to figure out:

1. Grouping the variables appropriately to one of the adapters
2. Turning those variables (represented as strings) into a dictionary
3. Further grouping those variables into a nested dictionary representing a single drive (in this crude example there are 2 drives per adapter)

Step 2 can easily be done by splitting on ":" and adding to an empty dictionary, but I'm having a hard time conceptualizing how to piece it all together.

The only thing I have right now is grouping the variables into groups of 5:

from itertools import zip_longest

def grouper(iterable, n, fillvalue=None):
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)
    
def get_drive_health():
    t = grouper(drives, 5)
    for i in t:
        print(i)

get_drive_health()

But this only groups every 5 elements in the list, which also includes "Adapter" which it shouldn't.

Answer 1

This might be a little fragile, but I hope the concept gets you started down a path that works for you. The answer is based on turning your data into an iterable so that we can conveniently call next() .

data = [
    'Adapter #0',
        'Enclosure Device ID: 252', 'Slot Number: 0', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up',
        'Enclosure Device ID: 252', 'Slot Number: 1', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up',
    'Adapter #1',
        'Enclosure Device ID: 252', 'Slot Number: 0', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up',
        'Enclosure Device ID: 252', 'Slot Number: 1', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up',
    'Adapter #2',
        'Enclosure Device ID: 252', 'Slot Number: 0', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up',
        'Enclosure Device ID: 252', 'Slot Number: 1', 'Media Error Count: 0', 'Other Error Count: 0', 'Firmware state: Online, Spun Up'
]

## ----------------------
## turn your data into an iterable so that we can call next()
## we could also have done this with a generator expression.
## ----------------------
data = iter(data)
## ----------------------

data_out = {}

## ----------------------
## Start by assuming/anticipating that the first item in data
## is the initial Adapter element. If it is not things should fail "ok-ish"
## ----------------------
adapter = next(data, None)
## ----------------------

while adapter and adapter.startswith("Adapter"):
    ## ----------------------
    ## Set the adapter id as the key of our output dictionary and
    ## initialize the value to be an empty dictionary if needs be
    ## ----------------------
    adapter_data = data_out.setdefault(adapter, {})
    ## ----------------------

    ## ----------------------
    ## fetch the next item in the list. We will anticipate that this is an
    ## enclosure but that is not strictly needed
    ## ----------------------
    next_data_point = next(data, None)
    ## ----------------------

    i = 0
    while next_data_point and next_data_point.startswith("Enclosure"):
        ## ----------------------
        ## next_data_point is an enclosure so add it to the dictionary
        ## ----------------------
        i += 1
        adapter_data[i] = {
            "enclosure_device_id": next_data_point.split(": ")[1],
            "slot_number": next(data).split(": ")[1],
            "media_error_count": next(data).split(": ")[1],
            "other_error_count": next(data).split(": ")[1],
            "firmware_state": next(data).split(": ")[1],
        }
        ## ----------------------

        ## ----------------------
        ## get the next value from the list
        ## if it is an Enclosure we keep in the inner loop
        ## ----------------------
        next_data_point = next(data, None)
        ## ----------------------

    ## ----------------------
    ## It was not an enclosure so pre-suppose it is an Adapter
    ## ----------------------
    adapter = next_data_point
    ## ----------------------

import json
print(json.dumps(data_out, indent=4))

This should give you:

{
    "Adapter #0": {
        "1": {
            "enclosure_device_id": "252",
            "slot_number": "0",
            "media_error_count": "0",
            "other_error_count": "0",
            "firmware_state": "Online, Spun Up"
        },
        "2": {
            "enclosure_device_id": "252",
            "slot_number": "1",
            "media_error_count": "0",
            "other_error_count": "0",
            "firmware_state": "Online, Spun Up"
        }
    },
    "Adapter #1": {
        "1": {
            "enclosure_device_id": "252",
            "slot_number": "0",
            "media_error_count": "0",
            "other_error_count": "0",
            "firmware_state": "Online, Spun Up"
        },
        "2": {
            "enclosure_device_id": "252",
            "slot_number": "1",
            "media_error_count": "0",
            "other_error_count": "0",
            "firmware_state": "Online, Spun Up"
        }
    },
    "Adapter #2": {
        "1": {
            "enclosure_device_id": "252",
            "slot_number": "0",
            "media_error_count": "0",
            "other_error_count": "0",
            "firmware_state": "Online, Spun Up"
        },
        "2": {
            "enclosure_device_id": "252",
            "slot_number": "1",
            "media_error_count": "0",
            "other_error_count": "0",
            "firmware_state": "Online, Spun Up"
        }
    }
}