优化OpenCL缓冲区写入？

Question

I am having a major bottleneck when writing to a buffer.

What I want to do is very simple. First of all, I am using two global ids (I am using an image2d). Each thread reads a 9 pixel values, the pixel at position (x,y) and its 8 neighbors, basically a 3x3 square block. This work is done by each thread. Now, I calculate some values and I want to write to the output buffer the results of each thread.

There are 64 values produced be each thread and I write them to the output buffer, so that means that the output buffer is of size (rows*cols*64). I also wanted to support calculations that support up to 640 values but obviously each thread writing 640 values to a buffer is impossible because of the VRAM required.

I must say that the threads write to different positions, there are no overwrites, that is there would be 64*number_of_threads = 64*global_id(0)*global_id(1) = 64*rows*cols values.

This is a major bottleneck in my code, I mean the writing of 64 values, I think it has to do with memory bandwidth but I am not so sure.

What can I do so that each thread can calculate and write 64 values to an output buffer efficiently? Is this not possible?

My GPU is rx 480 4gb, I know that the (rows*cols*64) size may sometimes be too big to fit the VRAM, but even if it fits, the writing is slow, I thought that the bandwidth was very high in gpus?

there are also other two output buffers, but their size is much smaller so we can ignore them.

In summary what this code does is this 1) read a square block of 9 pixels, where the middle one is the current value.

2) multiply the 8 neighbors with the current value, we get 8 values for each pixel.

3)write to the neighb buffer the 8 neighbors.

4) write 8*8 values to the Rx buffer. This buffer "simulates" the x_* x_^T result, that is a (8x1)x(1x8) matrix multiply of the neighbor values.

note that I am writing to the output buffers in a "transpose form", that is each thread at position (x,y) writes consecutively 64 values at (y,x), (y+1,x),...(y+63,x) this is because:

1) it is the fastest method! The version where I write as (x,y) -> (x+1,y),...(x+63,y) is definitely slower.

2) I need it in this form because I am using ArrayFire library which then needs to load the buffer, but it will consume the buffer in row-major order and put the contents inside its array in column-major order, that way there is no need to transpose the array (which will use a lot of vram copies)

Answer 1

First, as you haven't explicitly mentioned it, I'll point out that if you can, use your GPU manufacturer's profiling tools to verify your bottlenecks. Even if something seems like a bottleneck, it might be a red herring.

However, it does sound like global memory writes might be a problem in your kernel. As you don't give any specifics, I can only point out a few general things to watch out for:

1. Memory layout

It seems each work-item in your setup writes 64 values consecutively in memory. This means that every work-item will be writing to a different cache line, which is almost certainly not optimal. If you can change the memory layout of your output, try to arrange it such that work-items write to adjacent memory locations at the same time.

For example, you might currently have:

uint output_index = 64 * (get_global_size(0) * get_global_id(1) + get_global_id(0));
for (unsigned i = 0; i < 64; ++i)
{
    output[output_index + i] = calculation(inputs, i);
}

Here, work-item (0, 0) would first write to item 0, then item 1, then item 2, while work-item (1, 0) would first write to item 64, then 65, etc.

It is usually faster if work-item (0, 0) writes to index 0 at the same time as work-item (1, 0) writes to index 1, and so on. So if you can, try laying out your output array so that the value dimension has a higher-order stride, so you can write:

uint stride = get_global_size(0) * get_global_size(1);
uint output_index = (get_global_size(0) * get_global_id(1) + get_global_id(0));
for (unsigned i = 0; i < 64; ++i)
{
    output[output_index] = calculation(inputs, i);
    output_index += stride;
}

2. Using local memory as an intermediate

If changing the layout of global memory is not an option, you can instead write your results to local memory in an order that would be inefficient for global memory, and then copy it from local memory to global memory efficiently. By efficiently, I mean that adjacent work-items in your work-group should once again write to adjacent global memory locations. You can either do this explicitly or use the async_work_group_copy function in your kernel.

3. Compression

If there's some way to represent your 64 values in a way that's more space-efficient, that can help a lot, especially if you're subsequently sending the results back to the host CPU. For example, if precision is not so important and range is constrained and you are currently using float s, you could try using half (16-bit) floating-point values, or short / ushort 16-bit integer values for slightly higher precision but less range. Alternatively, if your values correlate in some way, you could use some other representation, such as a shared exponent.

4. Onward computation on GPU

If you are currently using the result from your computation on the host CPU, you will probably be bound by PCIe bandwidth, which is considerably lower than GPU-to-VRAM bandwidth. In this case, consider moving whatever further computation you are performing onto the GPU, even if the CPU implementation of this itself is not currently a bottleneck. Avoiding the copy from VRAM to system RAM may give you a bigger boost.

Better still, if you can avoid writing this result to global memory altogether, for example by performing the onward computation in the same kernel, possibly after storing the intermediate result in local memory to share it with the work-group, then you can avoid the memory bottleneck altogether.

5. Read the OpenCL optimisation guides

There might be other optimisations you can perform which are specific to your workload. As you haven't gone into any detail about what you're doing, we can't easily guess at those optimisations. The GPU manufacturers publish optimisation guides for OpenCL, make sure you read and understand them, and see if you can apply any of the advice to your task.

Answer 2

In my experience I have not identified any sort of bottleneck on writing to a buffer within a kernel, which I believed is what you are referring to.

Writing those 64 values in each thread should not be an issue your bottleneck might be somewhere else. It might be something that is being done before enqueueing the kernel, while preparing the buffer arguments.