Memory for 循环和算术运算中大型 numpy arrays 的使用问题

Question

I am quite new to python. I have a implicit time integration for loop, that is, next step depends on the previous step. Basically, I am trying to solve the time integration for all different cases by utilizing broadcast feature of numpy arrays without utilizing more for loops. Thus, I have large arrays.

The loop takes short time to run (2.5s) but while conducting the operations within the for loops my RAM (8 GB) reaches nearly its capacity. Therefore, there is a memory usage issue I am trying to overcome.

ndata = 9624
u.shape = (3, 9624, 11, 3, 1000)
p.shape = (9624, 11, 1, 1000)
a1.shape = (11, 3, 1000)
a2.shape = (11, 3, 1000)
a3.shape = (11, 3, 1000)
p_star.shape = (11, 3, 1000)
delta_t.shape =(11, 1, 1)
k_star.shape =(11, 3, 1000)

for i in range(0,ndata-1):
    p_star = p[i+1] + a1 * u[0,i] + a2 * u[1,i] + a3 * u[2,i]
    u[0,i+1] = p_star / k_star
    u[1,i+1] = gama * (u[0,i+1]-u[0,i]) / (beta * delta_t) + u[1,i] * (1 - gama/beta) + delta_t * u[2,i] * (1 - gama/(2*beta))
    u[2,i+1] = (u[0,i+1]-u[0,i]) / (beta * (delta_t ** 2)) - u[1,i] / (beta * delta_t) - u[2,i] * ((1 / (2*beta)) - 1)

You can see memory usage in the loop here

During the aritmatic operations, the temporary copies of the arrays are created and assigned. I think main issue here is creation the copies of the arrays. To overcome this issue, arithmatic operations should be conducted in-place.

To conduct these operations, as far as I know, optional argument out= of numpy ufuncs should be defined or += -= *= /= should be used. For example, numpy.sum(a1,a2, out=a1) .

But in order to conduct above mentioned operations in in-place fashion I have to define a lot of auxiliary (or buffer) variables. My question is what is the optimal way to decrease this memory usage. I believe that, in python, there must be a elegant way to do these type of arithmetic operations in-place.

I am open to other suggestions as well.

buff1 = np.zeros_like(u[0,0])
buff2 = np.zeros_like(u[0,0])
buff3 = np.zeros_like(u[0,0])
buff4 = np.zeros_like(u[0,0])
for i in range(0,ndata-1):
    # first line
    np.multiply(a1, u[0,i], out = buff1);
    np.multiply(a2, u[1,i], out = buff2);
    np.multiply(a3, u[2,i], out = buff3);
    buff1 += p[i+1];
    buff1 += buff2;
    buff1 += buff3;
    #second line
    np.divide(buff1, k_star, out = u[0,i+1])
    #third line
    np.subtract(u[0,i+1], u[0,i], out = buff1);
    buff1 *= gama
    buff1 /= beta
    buff1 /= delta_t
    np.divide(gama,beta, out = buff2)
    np.subtract(1, buff2, out=buff2)
    np.multiply(u[1,i], buff2, out=buff2)
    np.multiply(2, beta, out = buff3)
    np.divide(gama, buff3, out = buff3)
    np.subtract(1, buff3, out = buff3)
    np.multiply(u[2,i], buff3, out = buff3)
    np.multiply(delta_t, buff3, out = buff3)
    np.add(buff1, buff2, out = u[1,i+1])
    np.add(buff3, u[1,i+1], out = u[1,i+1])
    #last line goes like that
    

Answer 1

The problem is actually the size of the arrays p and u. Combined, they are made of roughly a billion floats, each represented with 64 bits. Or, in total, 8 GBs. I am afraid that there is not much that you can do about it if you need to keep all 9264 time steps -- of course the situation changes if you can keep only the two last time steps.