Removing nested brackets from a pandas dataframe?

Question

So I am trying to convert a.mat file into a dataframe in order to run some data analysis After converting it, I have a dataframe structure (see 1 ), but I have no idea how to remove the brackets from the objects in the dataframe. I have tried utilizing:

mdataframe['0'] = mdataframe['0'].str[0]

and

mdataframe['0'] = mdataframe['0'].str.get(0)

as an attempt to fix the 0th column to no avail. Any help and guidance would be appreciated.

Thank you!

Answer 1

Thank you for your question. It is indeed a very interesting subject. Personally, I have never seen a problem like yours; nevertheless, it is quite straightforward to solve your DataFrame conversion problem.

First of all, you need two steps:

1. Have a squashing function that will be applied for each entry in your table (ie, DataFrame). This function must act like a dimensional reducer. Since we don't know how many dimensions we are to expect in each cell of your table, this function has to be capable of calling itself (a recursive function).
2. apply the squashing function for each entry of your table, and return the converted table.

Therefore, by following steps 1 and 2, I have created a code snippet that generates a DataFrame similar to your example and squashes its cells accordingly.

---

Code Snippet

import numpy as np
import pandas as pd
from typing import Any, List
from numbers import Number

def generateDataFrameWithnestedListsInItsCells() -> pd.DataFrame:
    df = pd.DataFrame.from_records([[[["a"]]], [["b"]], [[["c"]]], [["b"]]])
    

    return df


def squashList(element:List[Number]) -> Any:
    
    array = np.asanyarray(element, dtype=list)
    
    array = np.ravel(element)
    
    while np.ndim(array) > 1:
        squashList(element)
    
    return array[0]


if "__main__" == __name__:
    
    df = generateDataFrameWithnestedListsInItsCells()
    
    df2 = df.applymap(squashList)
    
    

Notice that the df instance has your nested lists, while its converted form (ie, the df2) has its correct entries.

I hope that this example helps you in your research.

Sincerely,