Multithreaded CPU to GPU Using CUDA 7.0/VS2012 C++/CLI/OpenCV

Question

I have a base VS2012 C++/CLI application that pulls images from more than one webcam and then does object recognition on them using OpenCV. Currently, each webcam has its own thread. This works but I am not getting the frame rate I would like.

I would like to implement some of the code on my NVidia GPU. Thus, I want each of my CPU threads to be able to asynchronously call the GPU and perform a series of functions.

I am a GPU newbie so I am wondering which makes more sense:

1) locking access to the GPU to one CPU thread at a time; or

2) something where each CPU thread can make calls to the GPU and either only certain cores work on that thread (and other cores work on other threads); or

3) something where the jobs are cached.

If #2 is a good option, is there some sort of guide on how to do it? I would need to keep somethings in the GPU memory specific to each CPU thread.

Thanks for any guidance.

Answer 1

A Next step could be to use CUDA-Streams . So independent Threads can run concurrently on the GPU. But be carefull, with that it's getting more complex to have a look over the used Memory, Registers and Cores. That would mean you need additional code to avoid overallocating or running out of registers. Anytime keep in mind, that every resource on GPU is limited. For Kepler take a look in the Kepler Whitepaper on page 6 - 8 and in the Performance Guidelines in the CUDA Programming Guide on page 79 and 80.