Julia @parallel for return with return statement

Question

How can I write a parallel for loop in a function that returns for all workers as soon as a condition is met?

Ie something like this:

function test(n)
  @sync @parallel for i in 1:1000
    {... statement ...}
    if {condition}
      return test(n+1)
    end
  end
end

where all the workers stop working on the for loop and only the main process returns? (and the other processes start again working with the next for loop?)

Answer 1

The question seems like a basic pattern for doing "embarrassingly parallel" search tasks. The @parallel for construct is good for partitioning work, but doesn't have the break short-circuit logic for stopping as the for in single process flow.

To demonstrate how to do this in Julia consider a toy problem of finding the combination of a combination lock with several wheels. Each setting of a wheel can be checked for correctness with some method (taking a combodelay time - see code below). After the correct number for a wheel is found, the next wheel is searched. The high level pseudo code is like the snippet given in the OP question.

The following is running code (on 0.5 and 0.6) to do this. Some comments explain details, and the code is given in a single chunk for easy cut-and-paste.

# combination lock problem parameters
const wheel_max = 1000  # size of wheel
@everywhere const magic_number = [55,10,993]  # secret combination
const wheel_count = length(magic_number)  # number of wheels
const combodelay = 0.01 # delay time to check single combination

# parallel short-circuit parameters
const check_to_work_ratio = 160  # ratio to limit short-circuit overhead

function find_combo(wheel,combo=Int[])
  done = SharedArray{Int}(1)       # shared variable to hold if and what combo
  done[1] = 0                      #  succeded. 0 means not found yet
  # setup counters to limit parallel overhead
  @sync begin
    @everywhere global localdone = false
    @everywhere global checktime = 0.0
    @everywhere global worktime = 0.0
  end
  # do the parallel work
  @sync @parallel for i in 1:wheel_max
    global localdone
    global checktime
    global worktime
    # if not checking too much, look at shared variable
    if !localdone && check_to_work_ratio*checktime < worktime
      tic()
      localdone = done[1]>0
      checktime += toq()
    end
    # if no process found combo, check another combo
    if !localdone
      tic()
      sleep(combodelay) # simulated work delay, {..statement..} from OP
      if i==magic_number[wheel]    # {condition} from OP
        done[1] = i              
        localdone = true
      end
      worktime += toq()
    else
      break
    end
  end
  if done[1]>0 # check if shared variable indicates combo for wheel found
    push!(combo,done[1])
    return wheel<wheel_count ? find_combo(wheel+1,combo) : (combo,true)
  else
    return (combo,false)
  end
end

function find_combo_noparallel(wheel,combo=Int[])
  found = false
  i = 0
  for i in 1:wheel_max
    sleep(combodelay)
    if i==magic_number[wheel]
      found = true
      break
    end
  end
  if found
    push!(combo,i)
    return wheel<wheel_count ? 
      find_combo_noparallel(wheel+1,combo) : (combo,true)
  else
    return (combo,false)
  end
end

function find_combo_nostop(wheel,combo=Int[])
  done = SharedArray{Int}(1)
  done[1] = 0
  @sync @parallel for i in 1:wheel_max
    sleep(combodelay)
    if i==magic_number[wheel]
      done[1] = i
    end
  end
  if done[1]>0
    push!(combo,done[1])
    return wheel<wheel_count ? 
      find_combo_nostop(wheel+1,combo) : (combo,true)
  else
    return (combo,false)
  end
end

result = find_combo(1)
println("parallel with short-circuit stopping:       $result")
@assert result == (magic_number, true)

result = find_combo_noparallel(1)
println("single process with short-circuit stopping: $result")
@assert result == (magic_number, true)

result = find_combo_nostop(1)
println("parallel without short-circuit stopping:    $result")
@assert result == (magic_number, true)

println("\ntimings")

print("parallel with short-circuit stopping        ")
@time find_combo(1);
print("single process with short-circuit stopping  ")
@time find_combo_noparallel(1)
print("parallel without short-circuit stopping     ")
@time find_combo_nostop(1)

nothing

There could be better looking implementations, and some meta-programming can hide some of the short-circuit machinery. But this should be good as a start.

Results should look approximately like this:

parallel with short-circuit stopping:       ([55,10,993],true)
single process with short-circuit stopping: ([55,10,993],true)
parallel without short-circuit stopping:    ([55,10,993],true)

timings
parallel with short-circuit stopping          4.473687 seconds
single process with short-circuit stopping   11.963329 seconds
parallel without short-circuit stopping      11.316780 seconds

This is calculated for demonstration with 3 worker processes. Real problems should have many more processes and more work per process and then the benefits of short-circuiting will be evident.