如何连续获得最高和第二频繁值？

Question

Let' say I have a dataframe with 1 million rows and 30 columns. I want to add a column to the dataframe and the value is "the most frequent value of the previous 30 columns". I also want to add the "second most frequent value of the previous 30 columns"

I know that you can do df.mode(axis=1) for "the most frequent value of the previous 30 columns", but it is so slow.

Is there anyway to vectorize this so it could be fast?

Answer 1

df.mode(axis=1) is already vectorized. However, you may want to consider how it works. It needs to operate on each row independently, which means you would benefit from "row-major order" which is called C order in NumPy. A Pandas DataFrame is always column-major order, which means that getting 30 values to compute the mode for one row requires touching 30 pages of memory, which is not efficient.

So, try loading your data into a plain NumPy 2D array and see if that helps speed things up. It should.

I tried this on my 1.5 GHz laptop:

x = np.random.randint(0,5,(10000,30))
df = pd.DataFrame(x)
%timeit df.mode(axis=1)
%timeit scipy.stats.mode(x, axis=1)

The DataFrame way takes 6 seconds (!), whereas the SciPy (row-major) way takes 16 milliseconds for 10k rows. Even SciPy in column-major order is not much slower, which makes me think the Pandas version is less efficient than it could be.