Python make master dictonary with list of values from N dictionaries with identical entries but different values

Question

I have a server that spits out status updates five times a second. After some manipulation, my program creates a JSON dict with roughly 3000 entries. This is received every 200ms, so the JSON is exactly the same in terms of its entries, but the values change. I need to test some inputs and see exactly which entries change, for how long, etc.. This is a sample of the structure of the JSON dict I get:

 {
  "start-flag": 123456789,
  "packet_length": 3012,
  "id": 1253,
  "Subsystem_1": {
    "mode": 4,
    "is_active": 0,
    ...
    "voltage-on": 1},
  "Subsystem_2": {
    "local_temp": 25,
    "temp_error": 0,
    ...
    "collective_status_error": 0},
  ...
  "irbh_system_status": 0,
  "end-flag": 987654321}

You get the idea. But the values change often, and I would like to somehow record all the received dictionaries over a timeframe and make a master that looks like this:

 {
  "start-flag": [123456789, 123456789, 123456789, 123456789]
  "packet_length": [3012, 3012, 3012, 3012]
  "id": [1253, 1442, 2345, 2355]
  "Subsystem_1": {
    "mode": [4, 1, 1, 4]
    "is_active": [0, 1, 1, 0]
    ...
    "voltage-on": [1, 2.4, 2.4, 1]},
  "Subsystem_2": {
    "local_temp": [25, 28, 35, 100],
    "temp_error": [0, 0, 0, 1],
    ...
    "collective_status_error": [0, 0, 0, 1]},
  ...
  "irbh_system_status": [0, 0, 0, 0],
  "end-flag": [987654321, 987654321, 987654321, 987654321]}

This would be 4 different JSON dicts added together to make a master. Entries are the same, but the values change over time, and I would like to see them. I'd probably need this for ~600 dicts as i receive them quite often and would need a 2min ish sample (or more).

What's the best way to make this master dict while running? Ie a function that takes a dict when called and if it's the first makes it the master dict, but if it is not the first, then simply adds all values of all entries (including those of nested-dictionaries) to the master, and makes all values in the master lists? The values of the entries of the dictionaries received are always either int or float.

EDIT: I forgot to include this, as I just discovered this bug trying to solve this myself. Some of the nested dictionaries have the same entry names. In the above example this could be shown with both Subsystem_1 and Subsystem_2 having the "mode" and "is_active" entry, I just forgot to include such a field. These are two different entries for two different subsystems (thus why they're in different nested dicts), but have the same entry name in their respective dictionary. Sadly I can't edit these names as it's being sent from a black box computer I can only read from.

Leniks' solution works if I add a

d = my_stream_json_dict

# all the names of the first level nested dicts,
# assuming no second level nested dicts exist.
headers = ['Subsystem_1', 'Subsystem_2', ....]
for a in headers:
    for k,v in d[a].items() :
        master[k].append(v)

but doesn't take multiple nested dicts having the same entry name into account (since I didn't mention it, my bad,), and if the same entry exists in multiple nested dicts. it's populated multiple times.

If the modified solution is ran only a single time with only one dictionary, the top level entries are not considered (to be honest, that's not an issue for me as only the nested dictionaries (subsystems) matter), but although every entry should have only one value as only 1 dictionary was provided, entries that share the same name over N nested dicts have N values in each nested dict entry

Answer 1

Try this:

from collections import defaultdict

master = defaultdict(list)
for d in received_dictionaries :  # this is your data from the web
    for k,v in d.items() :
        master[k].append(v)