Pandas 三列 Dataframe 到三重字典列表

Question

I have the following Dataframe named Data:

ID	Labs	SampleName
1	lab1	banana
1	lab1	potato
1	lab2	kiwi
1	lab2	cellulose
2	lab1	NaCl
2	lab2	Cl2

I want to convert it to JSON, grouping Samplename by Labs and Labs by ID, so it looks like this:

{
   "Data":[
      {
         "ID":1,
         "Labs":[
            {
               "Lab_Name":"lab1",
               "SampleName":[
                  {
                     "Sample":"banana"
                  },
                  {
                     "Sample":"potato"
                  }
               ]
            },
            {
               "Lab_Name":"lab2",
               "SampleName":[
                  {
                     "Sample":"kiwi"
                  },
                  {
                     "Sample":"celulose"
                  }
               ]
            }
         ]
      },
      {
         "ID":2,
         "Labs":[
            {
               "Lab_Name":"lab1",
               "SampleName":[
                  {
                     "Sample":"NaCl"
                  }
               ]
            },
            {
               "Lab_Name":"lab2",
               "SampleName":[
                  {
                     "Sample":"NaCl"
                  },
                  {
                     "Sample":"NaCl"
                  }
               ]
            }
         ]
      }
   ]
}

To do so, I have tried:

newdata = pd.DataFrame(data, columns = ['ID','Labs'])
newjsonfile = newdata.groupby(newdata['ID'], as_index=False).agg(list).to_dict(orient='records')

This has succesfully got me the first part of the JSON file which is the Labs grouped by ID, however for the second part (grouping Samples by Labs) this wont work. Because of that I've tried to convert both into lists of dictionaries and append these dicts and lists as such:

# create a ID list
ID_list = data['ID'].tolist()
ID_list= list(dict.fromkeys(ID_list))
JSONdata = {'Data': []}

counter = 0

# for loop 
for i in ID_list:
    JSONdata['Data'].append({'ID': i})
    vari = data.loc[data['ID'] ==i]
    lab_list = vari['Labs'].tolist()
    for j in lab_list:
        JSONdata['Data'][counter].update('Lab_Name': j)
    counter += 1

This does work however I only get the last Lab name in each ID, as update overwrites the first Lab name. How can I achieve the wanted result? Any hints on how can "pythonize" my code further?

Answer 1

You can try:

df = df.rename(columns={'Labs': 'Lab_Name', 'SampleName': 'Sample'})
new_df = (df.groupby(['ID', 'Lab_Name']).apply(lambda x: x[['Sample']].to_dict(
    'records')).reset_index()).rename(columns={0: 'SampleName'})
result = (new_df.groupby(['ID']).apply(lambda x: x[['Lab_Name', 'SampleName']].to_dict(
    'records')).reset_index()).rename(columns={0: 'Labs'}).to_dict('records')

Answer 2

You can use the code below;

{'Data': df.groupby(['ID']).\
apply(lambda x:
         {'ID': x.iloc[0,0], 
          'Labs': x.groupby(['Labs']) \
              .apply(lambda y: 
                     {'Lab_Name': y.iloc[0,1],
                      'SampleName': y.groupby(['SampleName']) \
                          .apply(lambda z:
                                 {'Sample': z.iloc[0,2]}
                                ).values.tolist()
                     }
             
             ).values.tolist()
         }
     ).values.tolist()
}