在python中默认如何在'float32'模式下工作？

Question

I am working with 100s of big .csv file size of approx 500GB. It is becoming difficult to handle this much of big data so I want to reduce overhead by converting float64 into float32 . I had done it explicitly with astype('float32') but it is converting back to float64 after an arithmetic operation.

data.dtypes -> dtype('float64')

data32 = data.astype('float32')
data32.dtypes -> dtype('float32')

#after a float operation
data_f = data32*1.01       #######-> I want to keep data type float32 after operation
data_f.dtypes -> dtype('float64')

I want to keep the default result in 32-bit mode . I have found one similar question but it was asked almost a decade ago. I am hoping to get a fix for this so that I can set 32bit operation globally.

Answer 1

Use pandas to read the csv and specify the type of each columns ; use pandas to write back the csv.
You can also use numpy for (fast) arithmetic operations and specify the dtype float32 that is not available in pure python.