Scipy-大型稀疏矩阵相乘会导致分段错误？

Question

I have a CSR sparse matrix in scipy of size 444075 x 444075. I wish to multiply it by its transpose. However, when I do m * mT it causes a segmentation fault 11 error. Is this a memory issue, and if so, is there a way to allocate more memory to the program? Is there a clever workaround/hack using subroutines other routines from scipy to multiply a different way?

Answer 1

Scipy uses 8 bytes per non-zero integer in a sparse csr_matrix. With 4GB of available memory, and a matrix size of 444075x444075, no more than 0.272243315% of the matrix elements may be non-zero.

While your original matrix may contain only 3348026 non-zero elements, keep in mind that multiplying the matrix with its transpose may yield a more densely populated resultant matrix.

If you are running your application from a unix shell, the segmentation fault may well be due to the process being out of memory. If you're running from python shell, however, you should get a MemoryError .

The memory available to python is determined by how much physical RAM and virtual memory the operating system has available. You can check how much memory is available with psutil : print psutil.phymem_usage() .

Answer 2

Resolved the issue, turns out this is a memory problem. I ran the operation on another machine and received a MemoryIssue (whereas my machine gives a segfault), and when given more memory it turns into a "negative dimensions not allowed error" a long way into it, which I presume is an integer overflow in the calculation.