将所有数据帧列强制转换为float的最快方法 - pandas astype slow

Question

Is there a faster way to cast all columns of a pandas dataframe to a single type? This seems particularly slow:

df = df.apply(lambda x: x.astype(np.float64), axis=1)

I suspect there's not much I can do about it because of the memory allocation overhead of numpy.ndarray.astype .

I've also tried pd.to_numeric but it arbitrarily chooses to cast a few of my columns into int types instead.

Answer 1

No need for apply , just use DataFrame.astype directly.

df.astype(np.float64)

apply -ing is also going to give you a pretty bad performance hit.

Example

df = pd.DataFrame(np.arange(10**7).reshape(10**4, 10**3))

%timeit df.astype(np.float64)
1 loop, best of 3: 

%timeit df.apply(lambda x: x.astype(np.float64), )
1 loop, best of 3: 

%timeit df.apply(lambda x: x.astype(np.float64), )
1 loop, best of 3: 

Answer 2

One efficient way would be to work with array data and cast it back to a dataframe, like so -

pd.DataFrame(df.values.astype(np.float64))

Runtime test -

In [144]: df = pd.DataFrame(np.random.randint(11,99,(5000,5000)))

In [145]: %timeit df.astype(np.float64) # @Mitch's soln
10 loops, best of 3: 121 ms per loop

In [146]: %timeit pd.DataFrame(df.values.astype(np.float64))
10 loops, best of 3: 42.5 ms per loop

The casting back to dataframe wasn't that costly -

In [147]: %timeit df.values.astype(np.float64)
10 loops, best of 3: 42.3 ms per loop # Casting to dataframe costed 0.2ms