julia中的量子蒙特卡洛方法能否并行运算？

Question

This is my main code of parallel operation:

using Distributed
using SharedArrays
nprocs()
addprocs(7)

Now, I need to store a variable about time:

variable = SharedArray{ComplexF64, 3}(Dim, steps, paths)

Note that "steps" and "paths" denote time series and total number of trajectories, respectively. However, if i define this variable, i will meet with the out of memory probelm because Dim=10000, steps=600, and paths=1000, though i can use multiple kernels to achieve parallel operation. The code of parallel operation can be written as

@sync @distributed for path=1:paths
                       ...
                       variable[:,:,path] = matrix_var
end

Actually, this variable is not my final result, and the result is

final_var = sum(variable, dim=3)

, which represents the summation of all trajectories.

Thus, I want to deal with the out of memory problem and simultaneously use parallel operation. If i cast away the dimension of "paths" when i define this variable, the out of memory problem will vanish, but parallel operation becomes invaild. I hope that there are a solution to overcome it.

Answer 1

Seems that for each value of path you should create the variable locally rather than on huge array. Your code might look more or less like this:

final_vars = @distributed (append!) for path=1:paths
    #create local variable for a single step
    locvariable =  Array{ComplexF64, 2}(undef, Dim, steps)
    # at any time locvariable is at most in nprocs() copies
    # load data to locvariable specific to path and do your job
    final_var = sum(locvariable, dim=2)
    [final_var]   # in this way you will get a vector of arrays
end