Python：Pandas 读取 csv：在读取 csv 时向下转型

Question

I have the following problem. I want to read a large csv with million rows and hudnereds of columns. I want to downcast the dtypes for the columns. My approach is to read the csv and then downcasting it with pd.to_numeric() . I do not know the number of columns and their types. Is there any possibility to downcast while reading the csv so i do not have to touch the dataframe twice?

My current approch is:

import pandas as pd
df = pd.read_csv(filePath, delimiter=delimiter, memory_map=True,engine='c', low_memory=True)
for column in df:
    if is_numeric_dtype(df[column]):
       df[column] = pd.to_numeric(df[column], downcast='signed')
       df[column] = pd.to_numeric(df[column], downcast='float')

Thanks in advance!

Answer 1

If someone has the same problem, you can easily read the fist two lines, calulculate the dtypes and mapping your preferred dtypes over it and using it as dtype argument when reading the whole file:

Example:

import pandas as pd
df = pd.read_csv(filePath, delimiter=delimiter, nrows=2, low_memory=True, memory_map=True,engine='c')
mapdtypes = {'int64': 'int8', 'float64': 'float32'}
dtypes = list(df.dtypes.apply(str).replace(mapdtypes))
dtype = {key: value for (key, value) in enumerate(dtypes)}
df = pd.read_csv(filePath, delimiter=delimiter, memory_map=True,engine='c', low_memory=True, dtype=dtype)