Cython Numpy代碼不比純python快

Question

首先我知道在SO上有許多類似的主題問題，但經過一天的搜索，閱讀和測試后，我找不到解決方案。

我有一個python函數，它計算numpy ndarray（mxn）的成對相關性。 我原本只是在numpy中做這個，但函數也計算了倒數對（即計算矩陣的行A和B之間的相關性，它也計算了行B和A之間的相關性。）所以我拿了一個稍微不同的方法，對於大m的矩陣來說快兩倍（我的問題的實際大小是m~8000）。

這很好，但仍然有點慢，因為會有很多這樣的矩陣，並且要做這些都需要很長時間。 所以我開始研究cython作為一種加快速度的方法。 我從我讀到的內容中了解到，cython並不會真正加速numpy。 這是真的，還是我缺少的東西？

我認為下面的瓶頸是np.sqrt ， np.dot ，調用ndarray的.T方法和np.absolute 。 我見過人們使用libc.math sqrt替換np.sqrt，所以我想我的第一個問題是，我可以使用libc.math中其他方法的類似函數嗎？ 我擔心我完全和完全不熟悉C / C ++ / C＃或任何C系列語言，因此這種打字和cython業務對我來說是一個非常新的領域，如果原因/解決方案顯而易見，請道歉。

如果不這樣做，關於我能做些什么才能獲得一些性能提升的想法？

下面是我的pyx代碼，設置代碼和對pyx函數的調用。 我不知道它是否重要，但是當我調用python setup build_ext --inplace它可以工作，但有很多警告，我真的不明白。 這些也可能是我沒有看到提速的原因嗎？

任何幫助都非常感謝，並為超長的帖子感到抱歉。

setup.py

from distutils.core import setup
from distutils.extension import Extension
import numpy
from Cython.Distutils import build_ext


setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("calcBrownCombinedP", 
                            ["calcBrownCombinedP.pyx"], 
                            include_dirs=[numpy.get_include()])]
)

和設置的輸出：

>python setup.py build_ext --inplace

running build_ext
cythoning calcBrownCombinedP.pyx to calcBrownCombinedP.c
building 'calcBrownCombinedP' extension
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Anaconda\lib\site-packages\numpy\core\include -IC:\Anaconda\include -IC:\Anaconda\PC -c calcBrownCombinedP.c -o build\temp.win-amd64-2.7\Release\calcbrowncombinedp.o
In file included from C:\Anaconda\lib\site-packages\numpy\core\include/numpy/ndarraytypes.h:1728:0,
                 from C:\Anaconda\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:17,
                 from C:\Anaconda\lib\site-packages\numpy\core\include/numpy/arrayobject.h:15,
                 from calcBrownCombinedP.c:340:
C:\Anaconda\lib\site-packages\numpy\core\include/numpy/npy_deprecated_api.h:8:9: note: #pragma message: C:\Anaconda\lib\site-packages\numpy\core\include/numpy/npy_deprecated_api.h(8) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
calcBrownCombinedP.c: In function '__Pyx_RaiseTooManyValuesError':
calcBrownCombinedP.c:4473:18: warning: unknown conversion type character 'z' in format [-Wformat]
calcBrownCombinedP.c:4473:18: warning: too many arguments for format [-Wformat-extra-args]
calcBrownCombinedP.c: In function '__Pyx_RaiseNeedMoreValuesError':
calcBrownCombinedP.c:4479:18: warning: unknown conversion type character 'z' in format [-Wformat]
calcBrownCombinedP.c:4479:18: warning: format '%s' expects argument of type 'char *', but argument 3 has type 'Py_ssize_t' [-Wformat]
calcBrownCombinedP.c:4479:18: warning: too many arguments for format [-Wformat-extra-args]
In file included from C:\Anaconda\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:26:0,
                 from C:\Anaconda\lib\site-packages\numpy\core\include/numpy/arrayobject.h:15,
                 from calcBrownCombinedP.c:340:
calcBrownCombinedP.c: At top level:
C:\Anaconda\lib\site-packages\numpy\core\include/numpy/__multiarray_api.h:1594:1: warning: '_import_array' defined but not used [-Wunused-function]
In file included from C:\Anaconda\lib\site-packages\numpy\core\include/numpy/ufuncobject.h:311:0,
                 from calcBrownCombinedP.c:341:
C:\Anaconda\lib\site-packages\numpy\core\include/numpy/__ufunc_api.h:236:1: warning: '_import_umath' defined but not used [-Wunused-function]
writing build\temp.win-amd64-2.7\Release\calcBrownCombinedP.def
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -shared -s build\temp.win-amd64-2.7\Release\calcbrowncombinedp.o build\temp.win-amd64-2.7\Release\calcBrownCombinedP.def -LC:\Anaconda\libs -LC:\Anaconda\PCbuild\amd64 -lpython27 -lmsvcr90 -o C:\cygwin64\home\Davy\SNPsets\src\calcBrownCombinedP.pyd

pyx代碼 - ' calcBrownCombinedP.pyx '

import numpy as np
cimport numpy as np
from scipy import stats
DTYPE = np.int
ctypedef np.int_t DTYPE_t

def calcBrownCombinedP(np.ndarray genotypeArray):
    cdef int nSNPs, i
    cdef np.ndarray ms, datam, datass, d, rs, temp
    cdef float runningSum, sigmaSq, E, df 
    nSNPs = genotypeArray.shape[0]
    ms = genotypeArray.mean(axis=1)[(slice(None,None,None),None)]
    datam = genotypeArray - ms
    datass = np.sqrt(stats.ss(datam,axis=1)) 
    runningSum = 0
    for i in xrange(nSNPs):
        temp = np.dot(datam[i:],datam[i].T)
        d = (datass[i:]*datass[i])
        rs = temp / d
        rs = np.absolute(rs)[1:]
        runningSum += sum(rs*(3.25+(0.75*rs)))

    sigmaSq = 4*nSNPs+2*runningSum

    E = 2*nSNPs

    df = (2*(E*E))/sigmaSq

    runningSum = sigmaSq/(2*E)
    return runningSum

針對某些純python測試上面的代碼 - ' test.py '

import numpy as np
from scipy import stats
import random
import time
from calcBrownCombinedP import calcBrownCombinedP
from PycalcBrownCombinedP import PycalcBrownCombinedP

ms = [10,50,100,500,1000,5000]

for m in ms:
    print '---testing implentation with m = {0}---'.format(m)    
    genotypeArray = np.empty((m,20),dtype=int)

    for i in xrange(m):
        genotypeArray[i] = [random.randint(0,2) for j in xrange(20)] 

    print genotypeArray.shape 


    start = time.time()
    print calcBrownCombinedP(genotypeArray)
    print 'cython implementation took {0}'.format(time.time() - start)

    start = time.time()
    print PycalcBrownCombinedP(genotypeArray)
    print 'python implementation took {0}'.format(time.time() - start)

並且該代碼的輸出是：

---testing implentation with m = 10---
(10L, 20L)
2.13660168648
cython implementation took 0.000999927520752
2.13660167749
python implementation took 0.000999927520752
---testing implentation with m = 50---
(50L, 20L)
8.82721138
cython implementation took 0.00399994850159
8.82721130234
python implementation took 0.00500011444092
---testing implentation with m = 100---
(100L, 20L)
16.7438983917
cython implementation took 0.0139999389648
16.7438965333
python implementation took 0.0120000839233
---testing implentation with m = 500---
(500L, 20L)
80.5343856812
cython implementation took 0.183000087738
80.5343694046
python implementation took 0.161000013351
---testing implentation with m = 1000---
(1000L, 20L)
160.122573853
cython implementation took 0.615000009537
160.122491308
python implementation took 0.598000049591
---testing implentation with m = 5000---
(5000L, 20L)
799.813842773
cython implementation took 10.7159998417
799.813880445
python implementation took 11.2510001659

最后，純python實現' PycalcBrownCombinedP.py '

import numpy as np
from scipy import stats
def PycalcBrownCombinedP(genotypeArray):
    nSNPs = genotypeArray.shape[0]
    ms = genotypeArray.mean(axis=1)[(slice(None,None,None),None)]
    datam = genotypeArray - ms
    datass = np.sqrt(stats.ss(datam,axis=1)) 
    runningSum = 0
    for i in xrange(nSNPs):
        temp = np.dot(datam[i:],datam[i].T)
        d = (datass[i:]*datass[i])
        rs = temp / d
        rs = np.absolute(rs)[1:]
        runningSum += sum(rs*(3.25+(0.75*rs)))

    sigmaSq = 4*nSNPs+2*runningSum

    E = 2*nSNPs

    df = (2*(E*E))/sigmaSq

    runningSum = sigmaSq/(2*E)
    return runningSum

Answer 1

使用kernprof分析顯示瓶頸是循環的最后一行：

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
<snip>
    16      5000      6145280   1229.1     86.6          runningSum += sum(rs*(3.25+(0.75*rs)))

這並不奇怪，因為您在Python和Cython版本中都使用了Python內置函數sum 。 當輸入數組具有形狀(5000, 20) np.sum時，切換到np.sum代碼加速4.5倍。

如果精度損失很小，那么您可以利用線性代數進一步加速最后一行：

np.sum(rs * (3.25 + 0.75 * rs))

實際上是一個矢量點積，即

np.dot(rs, 3.25 + 0.75 * rs)

這仍然是次優的，因為它循環rs三次並構造兩個rs -sized臨時數組。 使用初等代數，可以將此表達式重寫為

3.25 * np.sum(rs) +  .75 * np.dot(rs, rs)

這不僅給出了原始結果而沒有前一版本中的舍入錯誤，而只是循環rs兩次並使用常量內存。（*）

瓶頸現在是np.dot ，所以安裝一個更好的BLAS庫會比在Cython中重寫整個東西更多。

（*）或最新NumPy中的對數內存，它具有np.sum的遞歸重新實現，比舊的迭代更快。