时间乘法矩阵和向量

Question

I wrote this algorithm that resolves matrix-vector product two ways: Imagine i have a matrix A nxn and a vector X nx1

1) inner-product: Y11 = A11*X11 + A12*X21 + .... + A1n*n1 and so on

2) lineal combination: Y = Colum1mn*X1 + Column2*X2 and so on

I did this because i wanted to compare who is the fastest way to multiply Matrix-Vector i tried with

n=[10000,9000,8000,7000,6000,5000,4000,3000,2000,1500,1000,800,500,300,100]

run the algorith 3 times and took the average value with every value of n and got this graph , and i was right, inner-product is the fastest because of how is saved matrix in memory thats ok but i ran algorithm 3 times to graph the points and got that Time of N=7000 > time of N=8000 > time of N=9000. And i want to know why is this? i thought that it could be something like the computer will start calculating slowly but if i run the algorithm more times it will calculate faster. But i ran the algorithm 7 times after the first 3 (10 in total) and got about the same result (this time n=7000 time was about 130 sec not 150 but still is time 7000 > time 8000 > time 9000)

This is the code i wrote

def Lineal_C(A,x,n):
    y=np.zeros(n)    
    t = time.clock()
    for j in range(n):
       for i in range(n):
          y[i]=y[i]+A[i][j]*x[j][0]
    time_spent = time.clock() - t
    print ("%.10f sec" % (time_spent)+" n"+str(n)+" Lineal Combination ")

def Inner_P(A,x,n):
    y=np.zeros(n)
    t = time.clock()
    for i in range(0,n):
        for j in range(n):
            y[i]=y[i]+A[i][j]*x[j][0]

    time_spent = time.clock() - t
    print ("%.10f sec" % (time_spent)+" n="+str(n)+" ------MatVectFila - Producto Interno--------")

Answer 1

At first I thought the answer might involve some important detail pertaining to the actual size of the data and differences in execution paths based on size, but at the end of the day the loops are in Python. To the best of my understanding, the standard Python interpreter doesn't do anything fancy (eg, no JIT).

The important part of the linked gist is that all the calculations are being performed on separate, non-blocking threads. Even though the sizes are specified from largest to smallest, the printouts are reported from smallest to largest. This is because the code starts all the threads in parallel (well, in rapid succession) and the smallest ones finish first (despite starting slightly later). This means that eventually only the large data threads will be running, and they will be relatively faster towards the end.

If you run each data size sequentially, then you never see these strange timing effects because each size is no longer "fighting for cycles" with the other sizes. Note, this may be actually fighting for cycles in the processor or fighting for interpretation time (wording?) with the global interpreter lock.

The code below forces each size to complete before running the next size:

    for value in n_values:
       t1 = threading.Thread(target=threadss , args=(value,))
       t1.start()
       t1.join() #forces t1 to finish before the next loop iteration

At this point I thought I had figured out the problem in that larger sizes were executing later, and thus could run relatively faster since they didn't have to share processing time with as many threads. However, if they start at the same time, running faster for just a bit shouldn't make something take less time overall, just less time per unit size (which is not what was measured). For example, if you start n=7000 and n=8000 nearly simultaneously, the n=8000 should start to compute slightly faster once n=7000 has finished, but it shouldn't execute faster than n=7000. So, something was missing.

Interestingly, if you add a print statement to the beginning of the loops, you should notice that the loop threads execute nearly sequentially from small to large thread sizes. At first I though this was something strange with the threading process, but it turns out the real culprit is this line:

matrix, vector = ranMatrix(value)

That function call builds up a list of lists of numbers, something that is hugely inefficient in Python as compared to something like numpy - so this takes a non-neglible amount of time (which is likely exacerbated by Python's Global Interpreter Lock). This call is outside of the timer and has the effect of delaying the start of larger data loops until after smaller ones have finished (again, visible if you print at the start of the loops). Thus, my assumption of all the loops starting at the same time is incorrect, and in reality large data loops don't start until after smaller data loops because of this initialization that isn't timed. Therefore I think that my above conclusion is correct, I just hadn't realized the importance of the delayed start.

So to reiterate, when 7000 is running, 8000, 9000, and 10000 are still building their arrays. When 8000 is running, 7000 is done (as are the lower sizes), and only 9000 and 10000 are building their arrays. At some point it appears that the faster execution per data point versus the larger array/matrix size tips in favor of the faster execution, meaning that overall the execution time drops. Put another way, if total time is n_elements*time_per_element, if time_per_element gets fast enough then total_time will decrease even if n_elements has increased.

In addition to blocking threads, you can also initialize all the data up front. The two changes are:

    matrix, vector = ranMatrix(10000)
    for value in n_values:
       t1 = threading.Thread(target=threadss , args=(value,matrix,vector))
       t1.start()

and

def threadss(value,matrix,vector):

    threading.Thread(target=linealComb , args=(matrix , vector, value,)).start()
    threading.Thread(target=innerProduct , args=(matrix , vector, value,)).start()

Since the code runs over a fixed size, it doesn't matter if the matrix and vector are actually larger. This too also makes it so that larger data sizes don't ever run faster.