I've searched stackoverflow for a solution to this -> but all solutions are slightly different to my needs.
I have a large ndarray (roughly 107 million rows) lets call it df
:
[{'A': 5, 'C': 3, 'D': 3},
{'A': 7, 'B': 9, 'F': 5},
{'B': 4, 'C': 7, 'E': 6}]
I need it to be converted to a DataFrame as time efficiently as possible. This is an example desired output:
A B C D E F
0 5.0 NaN 3.0 3.0 NaN NaN
1 7.0 9.0 NaN NaN NaN 5.0
2 NaN 4.0 7.0 NaN 6.0 NaN
I have tried pd.DataFrame(df)
and pd.DataFrame.from_dict(df)
but these give me the output:
0
0 {'A': 5, 'C': 3, 'D': 3}
1 {'A': 7, 'B': 9, 'F': 5}
2 {'B': 4, 'C': 7, 'E': 6}
The question: How do I convert df
to the desired output?
EDIT:
I have tried anky_91's solution. This will work for a list - NOT an ndarray. I want to avoid converting to a list as having 107million values in a list causes memory errors.
pd.DataFrame(df).sort_index(axis=1)
This still gives me the same output as pd.DataFrame(df). It outputs a DataFrame containing one column with dictionary in each row.
I think input data are different:
L = [[{'A': 5, 'C': 3, 'D': 3}],
[{'A': 7, 'B': 9, 'F': 5}],
[{'B': 4, 'C': 7, 'E': 6}]]
print (pd.DataFrame(L))
0
0 {'A': 5, 'C': 3, 'D': 3}
1 {'A': 7, 'B': 9, 'F': 5}
2 {'B': 4, 'C': 7, 'E': 6}
Possible solution is flattening:
from itertools import chain
df = pd.DataFrame(chain.from_iterable(L)).sort_index(axis=1)
print (df)
A B C D E F
0 5.0 NaN 3.0 3.0 NaN NaN
1 7.0 9.0 NaN NaN NaN 5.0
2 NaN 4.0 7.0 NaN 6.0 NaN
If input datais numpy array use solution from comment by @Code Different:
arr = np.array([{'A': 5, 'C': 3, 'D': 3},
{'A': 7, 'B': 9, 'F': 5},
{'B': 4, 'C': 7, 'E': 6}])
df = pd.DataFrame(arr.tolist()).sort_index(axis=1)
print (df)
A B C D E F
0 5.0 NaN 3.0 3.0 NaN NaN
1 7.0 9.0 NaN NaN NaN 5.0
2 NaN 4.0 7.0 NaN 6.0 NaN
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.