OpenMP迭代for循环并行区域

Question

Sorry if the title's a big unclear. I don't quite know how to word this.

I'm wondering if there's any way I can do the following:

#pragma omp parallel
{
    for (int i = 0; i < iterations; i++) {
        #pragma omp for
        for (int j = 0; j < N; j++)
            // Do something
    }
}

Ignoring things such as omitting private specifiers in the for loop, is there any way that I can fork threads outside of my outer loop so that I can just parallelize the inner loop? From my understanding (please do correct me if I'm wrong), all threads will execute the outer loop. I'm unsure about the behavior of the inner loop, but I think the for will distribute chunks to each threads that encounter it.

What I want to do is not have to fork/join iterations times but just do it once in the outer loop. Is this the right strategy to do so?

What if there were another outer loop that shouldn't be parallelized? That is...

#pragma omp parallel
{

    for (int i = 0; i < iterations; i++) {
        for(int k = 0; k < innerIterations; k++) {
            #pragma omp for
            for (int j = 0; j < N; j++)
                // Do something

            // Do something else
        }
    }
}

It'd be great if someone were to point me to an example of a large application parallelized using OpenMP so that I could better understand strategies to employ when using OpenMP. I can't seem to find any.

Clarification: I'm looking for solutions that do not change loop ordering or involve blocking, caching, and general performance considerations. I want to understand how this could be done in OpenMP on the loop structure as specified. The // Do something may or may not have dependencies, assume that they do and that you can't move things around.

Answer 1

The way you handled the two for loops looks right to me, in the sense that it achieves the behavior you wanted: the outer loop is not parallelized, while the inner loop is.

To better clarify what happens, I'll try to add some notes to your code:

#pragma omp parallel
{
  // Here you have a certain number of threads, let's say M
  for (int i = 0; i < iterations; i++) {
        // Each thread enters this region and executes all the iterations 
        // from i = 0 to i < iterations. Note that i is a private variable.
        #pragma omp for
        for (int j = 0; j < N; j++) {
            // What happens here is shared among threads so,
            // according to the scheduling you choose, each thread
            // will execute a particular portion of your N iterations
        } // IMPLICIT BARRIER             
  }
}

The implicit barrier is a point of synchronization where threads wait for each other. As a general rule of the thumb it is thus preferable to parallelize outer loops rather than inner loops , as this will create a single point of synchronization for the iterations*N iterations (instead of the iterations points you are creating above).

Answer 2

I'm not sure I can answer your question. I have only been using OpenMP for a few months now but when I try to answer questions like this I do some hello world printf tests like I show below. I think that may help answer your questions. Also try #pragma omp for nowait and see what happens.

Just make sure when you "// Do something and // Do something else" that you don't write to the same memory address and create a race condition. Also, if you're doing a lot of reading and writing you need to think about how to efficiently use the cache.

#include "stdio.h"
#include <omp.h>
void loop(const int iterations, const int N) {
    #pragma omp parallel
    {
        int start_thread = omp_get_thread_num();
        printf("start thread %d\n", start_thread);
        for (int i = 0; i < iterations; i++) {
            printf("\titeration %d, thread num %d\n", i, omp_get_thread_num());
            #pragma omp for
            for (int j = 0; j < N; j++) {
                printf("\t\t inner loop %d, thread num %d\n", j, omp_get_thread_num());
            }
        }
    }
}

int main() {
    loop(2,30);
}

In terms of performance you might want to consider fusing your loop like this.

#pragma omp for
for(int n=0; n<iterations*N; n++) {
    int i = n/N;
    int j = n%N;    
    //do something as function of index i and j
}

Answer 3

It is difficult to answer since it really depends on the dependencies inside your code. But a general way to solve this is to invert the nesting of the loops, like this:

#pragma omp parallel
{
    #pragma omp for
    for (int j = 0; j < N; j++) {
        for (int i = 0; i < iterations; i++) {
            // Do something
        }
    }
}

Off course, this can or cannot be possible, depending of what is your code inside the loop.