如何根据时间戳同步或合并 dataframe 与 json 数据

Question

there are many examples out there on how to merge two pandas dataframe but my Problem is that I have two kind of data. data1 is a csv data that I read it with pandas and turn it to dataframe and the other data2 is in json format.

here is an example of the json data:

[{'timestamp': 1572430625231, 'url': 'brakePressure', 'value': 10},
 {'timestamp': 1572430625275, 'url': 'lateralAcceleration', 'value': 120},
 {'timestamp': 1572430625290, 'url': 'longitudinalAcceleration', 'value': 110},
 {'timestamp': 1572430625299, 'url': 'acceleratorPosition', 'value': 1230},
 {'timestamp': 1572430625310, 'url': 'currentTorque', 'value': 10}]

as you can see every feature value is inside a dictionary with a timestamp. the problem is if I convert this to a dataframe the rows would be a timestamp, url and value but I don't want that, I want that my columns (features) would be brakePressure, lateralAcceleration etc.. and in every column there are all values that correspond to that feature and that are inside this json.

my Goal is to merge the two datasets based on timestamp. this is so hard because here in the json I have a timestamp assosiated with every single feature value, on the contrary in the csv data I have a timestamp that correspond to a row (that mean n feature value and not a single one). I tried so hard to do this but no chance, so I thought maybe I can search which is the closest timestamp and then replace a single value at a time, here is my try:

def sync_vehicle_gps_data(dataset=vehicle_data, gps_data=gps_data):
    vehicle = dataset.copy()
    gps = gps_data.copy()
    d = {}

    for json in vehicle:

        timestamp, feature, val = json.values()
        index = abs(gps['timestamp'] - timestamp).idxmin()
        print("closest value index = ", index)
        gps.at[index, feature] = val

    return gps

the vehicle_data are the json data and the gps_data is a pandas DataFrame, as you can see I search throught the whole dataset which timestamp is the closest to the single feature timestamp and then I update that specific value but this didn't work well for me. I ended up having messed up data. Is there any way to do this in python? I can also use any other libraries if there is one so I'm not restricted to pandas.

the expected output is that I can append those values in the json above to the existing dataframe so that means in this example that a new columns 'brakePressure', 'lateralAcceleraiton' etc.. would be added and the value of each feature(as it is in the json above, yes they are zeros but it's only an example) would be put in the row where the timestamp of that row is the nearest to the timestamp of the timestamp key of each feature in the json above. I know it is a very complex problem, it's not easy to explain but I hope you understand what I mean. here is an example: let's say this is the gps data

      timestamp        X      Y     Z 
      1572430510880  595.00  179.00 -60.00
      1572430510890   -0.23   -0.09   0.01
      1572430510900   -0.11   -0.02   0.04
      1572430510910   -1.96   -5.19  -6.10

I want this output ( I ll show this only for one feature):

      timestamp        X      Y        Z    brakePressure
      1572430510880  595.00  179.00 -60.00   10
      1572430510890   -0.23   -0.09   0.01   nan
      1572430510900   -0.11   -0.02   0.04   nan
      1572430510910   -1.96   -5.19  -6.10   nan

the value of the brakePressure feature in the dictionary was put in the first row because the closest gps timestamp to the timestamp of the brakePressure in the dictionary above is also in the first row. Basically I want to do that same thing for all the features in the json, I want to synchronize all those feature values with the gps data

Answer 1

Ref: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html

You can create 0's and 1's as column values:

import json

json_data = [{}, ]
df1 = pd.read_json(json.dumps(json_data))

which gives

                timestamp                       url  value
0 2019-10-30 10:17:05.231             brakePressure     10
1 2019-10-30 10:17:05.275       lateralAcceleration    120
2 2019-10-30 10:17:05.290  longitudinalAcceleration    110
3 2019-10-30 10:17:05.299       acceleratorPosition   1230
4 2019-10-30 10:17:05.310             currentTorque     10

And, then

ready_to_join_df = pd.get_dummies(df1, prefix="", prefix_sep="")

which results in:

                timestamp  value  acceleratorPosition  brakePressure  currentTorque  lateralAcceleration  longitudinalAcceleration
0 2019-10-30 10:17:05.231     10                    0              1              0                    0                         0
1 2019-10-30 10:17:05.275    120                    0              0              0                    1                         0
2 2019-10-30 10:17:05.290    110                    0              0              0                    0                         1
3 2019-10-30 10:17:05.299   1230                    1              0              0                    0                         0
4 2019-10-30 10:17:05.310     10                    0              0              1                    0                         0

And now you can join two dataframes on key timestamp.