Numpy dot product of very large arrays

Question

I have a problem of very slow running code. A try to calculate the dot product of a very large binary matrix (170544 X 22) by its transpose. First I tried this code

import numpy as np
start = time.time()
import warnings
warnings.filterwarnings("ignore", message='genfromtxt',     category=UserWarning)
np.set_printoptions(threshold=np.nan)

fin = open('E:/myscripts/Abin.txt', 'rb')    # input file (170544X22     binary matrix)
fin1 = open('E:/myscripts/AbinT.txt', 'rb')    # input file (22X170544        binary matrix the transpose of Abin)
fout = open('E:/myscripts/RPartial.txt', 'w+b')  # output file

FW = np.genfromtxt(fin,delimiter=',',   dtype=int)
WF = np.genfromtxt(fin1,delimiter=',',   dtype=int)

r = np.dot(FW,WF) #r calculation
np.savetxt(fout, r,  fmt='%i' ,delimiter=',', newline='\r\n')

fin.close()
fin1.close()
fout.close() 

but there was a memory error. Then I changed the r calculation using a row approach:

for row in FW:
    a=FW[row,:]
    r = np.dot(a,WF)
    np.savetxt(fout, r,  fmt='%i' ,delimiter=',', newline='\r\n')

The code now works, but its is very slow, after 90 min only 8000 rows was calculated. The hardware is a I7 with 12GB of ram, running windows 64 bits. How a can speed up the code?

The data is like this

([1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0],
 [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,0,0,0,0,0,0],
 [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,0,0,0,0,0],
 [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,1,0,0,0,0],
 [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,0,0,0])

and the result should be like

([15,14,14,14,14],
 [14,15,14,14,14],
 [14,14,15,14,14],
 [14,14,14,15,14],
 [14,14,14,14,15]) .

Answer 1

This sounds like a sparse matrix problem for which scipy provides a package . These are matrices with many 0 elements (like in your example). Operations will take the sparseness into account and the matrix will take up less space in memory. Remember to do matrix operations like FW.dot(WF) (use this rather than np.dot ).

Answer 2

As I have written in the comments the output array will be 232GB in size(int64). If you wan't to store the results to disk using h5py for this task would be a proper solution.

A summation about first axis simplifies the problem a bit. If you wan't the pure dot product I can update my answer. But this will be a bit more complicated and slower.

res=np.zeros(WF.shape[1])
for i in range(WF.shape[1]):
  a=np.copy(WF[:,i])
  r=np.dot(FW,a)
  res[i] = np.sum(r)