从 csv 文件中读取 Pandas dataframe 并转换为 ZA7F5F35426B927411173231B5632 类型

Question

I want to read a Pandas dataframe with elements of particular python types, such as arrays and dictionaries, and numpy arrays. I want to read it such that I can immediately work with them (now they are read as a string). How do I do that?

I want functionality similar to ast.literal_eval , but hopefully there is a way to do it without looping over the whole dataframe.

Edit: as requested, a minimal reproducible example.

import pandas as pd
import numpy as np

output = pd.DataFrame()
data = {'integer':1, 'list': [1,2,3], 'dictionary':{}, 'np_arrar' = np.array([1,2,3]}
output = output.append({}, ignore_index=True)
    
filename = 'data.csv'
output.to_csv(filename)

input_data = pd.read_csv(filename, ???) # What to do here?

Ideally, I want a way where I don't have to input the datatypes manually (not sure if there is such approach).

Answer 1

For people of the future: for simple data types it is possible to use the dtype parameter, like so

input_data = pd.read_csv(filename, dtype = {'integer':'int'})

However, for objects, this does not work properly. Then you can use the converters parameter instead. This is a dictionary of functions to convert a certain column in your data. One can use the function ast.literal_eval from ast

input_data = pd.read_csv(filename, converters= {'integer': ast.literal_eval, 'dictionary': ast.literal_eval, 'list': ast.literal_eval}

Be careful though, this does not work with numpy arrays as you will encounter an error SyntaxError: invalid syntax because numpy arrays are stored without commas, this is not valid Python Syntax. Instead you can define your own function

def string_to_numpyArray(x):
    return np.fromstring(x[1:-1],dtype = float, sep = ' ')

and then use this as follows

input_data = pd.read_csv(filename, converters= {'integer': ast.literal_eval, 'dictionary': ast.literal_eval, 'list': ast.literal_eval, 'np_list':string_to_numpyArray}

Hope this is helpful for someone.

Cheers