如何比较两个字典并找到匹配值

Question

I'm pulling data from an API for a weather system. The API returns a single JSON object with sensors broken up into two sub-nodes for each sensor. I'm trying to associate two (or more) sensors with their time-stamps. Unfortunately, not every sensor polls every single time (although they're supposed to).

In effect, I have a JSON object that looks like this:

{
    "sensor_data": {
        "mbar": [{
            "value": 1012,
            "timestamp": "2019-10-31T00:15:00"
        }, {
            "value": 1011,
            "timestamp": "2019-10-31T00:30:00"
        }, {
            "value": 1010,
            "timestamp": "2019-10-31T00:45:00"
        }],
        "temperature": [{
            "value": 10.3,
            "timestamp": "2019-10-31T00:15:00"
        }, {
            "value": 10.2,
            "timestamp": "2019-10-31T00:30:00"
        }, {
            "value": 10.0,
            "timestamp": "2019-10-31T00:45:00"
        }, {
            "value": 9.8,
            "timestamp": "2019-10-31T01:00:00"
        }]
    }
}

This examples shows I have one extra temperature reading, and this example is a really small one.

How can I take this data and associate a single reading for each timestamp, gathering as much sensor data as I can pull from matching timestamps? Ultimately, I want to export the data into a CSV file, with each row representing a slice in time from the sensor, to be graphed or further analyzed after.

For lists that are exactly the same length, I have a solution:

sensor_id = '007_OHMSS'
sensor_data = read_json('sensor_data.json') # wrapper function for open and load json
list_a = sensor_data['mbar']
list_b = sensor_data['temperature']

pair_perfect_sensor_list(sensor_id, list_a, list_b)
def pair_perfect_sensor_lists(sensor_id, list_a, list_b):
    # in this case, list a will be mbar, list_b will be temperature
    matches = list()
    if len(list_a) == len(list_b):
        for idx, reading in enumerate(list_a):
            mbar_value = reading['value']
            timestamp = reading['timestamp']
            t_reading = list_b[idx]
            t_time = t_reading['timestamp']
            temp_value = t_reading['value']
            print(t_time == timestamp)

            if t_time == timestamp:
                match = {
                    'sensor_id': sensor_id,
                    'mbar_index': idx,
                    'time_index': idx,
                    'mbar_value': mbar_value,
                    'temp_value': temp_value,
                    'mbar_time': timestamp,
                    'temp_time': t_time,
                }
                print('here is your match:')
                print(match)
                matches.append(match)
            else:
                print("IMPERFECT!")
                print(t_time)
                print(timestamp)
        return matches
    return failure

When there's not a match, I want to skip a reading for the missing sensor (in this case, the last mbar reading) and just do an N/A.

In most cases, the offset is just one node - meaning temp has one extra reading, somewhere in the middle.

I was using the idx index to optimize the speed of the process, so I don't have to loop through the second (or third, or nth) dict to see if the timestamp exists in it, but I know that's not preferred either, because dicts aren't ordered. In this case, it appears every sub-node sensor dict is ordered by timestamp, so I was trying to leverage that convenience.

Is this a common problem? If so, just point me to the terminology. But I've searched already and cannot find a reasonable, efficient answer besides "loop through each sub-dict and look for a match".

Open to any ideas, because I'll have to do this often, and on large (25 MB files or larger, sometimes) JSON objects. The full dump is up and over 300 MB, but I've sliced them up by sensor IDs so they're more manageable.

Answer 1

You can use.get to avoid type errors to get an output like this.

st=yourjsonabove

mbar={}
for item in st['sensor_data']['mbar']: 
    mbar[item['timestamp']] = item['value']

temperature={}
for item in st['sensor_data']['temperature']:
    temperature[item['timestamp']] = item['value']

for timestamp in temperature: 
   print("Timestamp:" , timestamp, "Sensor Reading: ", mbar.get(timestamp), "Temperature Reading: ", temperature[timestamp]) 

leading to output:

Timestamp: 2019-10-31T00:15:00 Sensor Reading:  1012 Temperature Reading:  10.3
Timestamp: 2019-10-31T00:30:00 Sensor Reading:  1011 Temperature Reading:  10.2
Timestamp: 2019-10-31T00:45:00 Sensor Reading:  1010 Temperature Reading:  10.0
Timestamp: 2019-10-31T01:00:00 Sensor Reading:  None Temperature Reading:  9.8

Does that help?

Answer 2

Try like below.

dict1 = {'red':[1,2,3],'blue':[2,3,4],'orange':[3,4,5]}

dict2 = {'green':[3,4,5],'yellow':[2,3,4],'red':[5,2,6]}

matches = []
for key1 in dict1:
    for key2 in dict2:
        if dict1[key1] == dict2[key2]:
            matches.append((key1, key2))

print(matches)

Output will be

[('blue', 'yellow'), ('orange', 'green')]

Answer 3

You could make a dict with timestamp keys of your sensor readings like

mbar = {s['timestamp']:s['value'] for s in sensor_data['mbar']}
temp = {s['timestamp']:s['value'] for s in sensor_data['temperature']}

Now it is easy to compare using the difference of the key sets

mbar.keys() - temp.keys()
temp.keys() - mbar.keys()