对于相同的数据，为什么pandas.DataFrame.cov（）方法比numpy.dot（…）快几个数量级？

Question

I was calculating covariance run times like this in ipython

>>> from pandas import DataFrame
>>> import numpy as np
>>> # create data frame set
>>> df = get_data()
>>> df.shape
(4795, 1000)
>>> %timeit df.cov()
10 loops, best of 3: 99.5 ms per loop
>>> mat = np.matrix(df.values)
>>> %timeit np.dot(mat.transpose(), mat)
1 loops, best of 3: 1min per loop

Answer 1

So, I've figured out the why of the observed speed difference. . . but not the why of the why. I'll update when I find that.

This is the answer to: "Why is the DataFrame.cov method so much faster than converting to a numpy matrix and using the np.cov or np.dot method?"

The DataFrame data type was int64. When it was converted to a numpy matrix using

mat = np.matrix(df.to_matrix())

The resulting 'mat' object is also of type int64.

Under the hood, the DataFrame.cov method converts its matrix to float64 before calling numpy's covariance method.

When running timeit's on numpy ndarrays or matrices of dtype int64, you see the same performance lag. On my machine with a dataset of shape (16497, 5000) int64 operations do not complete and sometimes crash with memory errors. float64 completes in seconds.

So, the short answer is the reason the numpy.dot method above is slower than the DataFrame.cov one is the datatypes were different.

I'm going to investigate why this gap exists.