pandas to_numeric(..., downcast='float') 失去精度

Question

Downcasting pandas dataframe (by columns) from float64 to float32 results in losing precision even though largest(9.761140e+02) and smallest (0.000000e+00) element is suitable for float32.

Dataset is pretty large, 55 million rows times 12 columns. This is the mean of the particular column without downcasting (1.343987e+00) and after is this 1.224472e+00.

Same results I am getting with np.astype() .

Answer 1

This was a pretty interesting question. I tested several dataframes starting from 1 million records to 55 million, the same size as yours, keeping min , max value similar to the ones you have.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

x, y = [], []
for idx, num in enumerate(range(1, 57, 2)):
    print(f"{idx+1}) Testing with {num} million records...")
    rows = num*(10**6)
    cols = ['col']

    df = pd.DataFrame(np.random.uniform(0, 9.761140e+02, size=(rows, len(cols))), columns=cols)
    df['col1'] = pd.to_numeric(df['col'], downcast='float')
    df['diff'] = df['col'] - df['col1']

    diff = df['col'].mean() - df['col1'].mean()

    x.append(num)
    y.append(diff)

plt.plot(x, y, 'ro')
plt.xlabel('number of rows (millions)')
plt.ylabel('precision value lost')
plt.show()

Here's the plot.

Based on the plot, it seems like, after 35 million records, there is a sudden increase in loss of precision and appears to be logarithmic in nature. I haven't figured out yet why it is the way it is.