Transfer OpenGL image on GPU from C++ to Python for deep learning

Question

I built a simulator in C++ with a pybind11 interface to run deep learning in Python using PyTorch. At each time step, I draw certain things from the simulator's scene using the SFML library (wrapper around openGL). I draw that on a texture, then get the pixels from that texture as follows:

glBindTexture(GL_TEXTURE_2D, imageTexture.getTexture().getNativeHandle());
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE, img.data());

I then move the pixels vector img from C++ to Python using the pybind11 interface. Problem is that this GPU-to-CPU operation is very slow. Since the vector in Python is then transferred back to the GPU for fast deep learning (CNN) operations, I was wondering how I could avoid that step.

My best guess so far is that I should at each step bind the texture in C++ (as in the code above), then right after that in the Python get the bound texture using CUDA, while keeping it on the GPU. However I couldn't figure out how to do that, I don't know much about GPUs and how CUDA/OpenGL work. Pointers to the right direction would be very appreciated!

Answer 1

You should use PBO(Pixel Buffer Object) for this operation.

Data transferring operation is very fast using PBO

https://www.khronos.org/opengl/wiki/Pixel_Buffer_Object

GLuint w_pbo[2];

 // Create pbo objects and than

 // Do your drawings.

int w_readIndex = 0;
int w_writeIndex = 1;
glReadBuffer(GL_COLOR_ATTACHMENT0);
w_writeIndex = (w_writeIndex + 1) % 2;
w_readIndex = (w_readIndex + 1) % 2;
glBindBuffer(GL_PIXEL_PACK_BUFFER, w_pbo[w_writeIndex]);
// copy from framebuffer to PBO asynchronously. it will be ready in the NEXT frame
glReadPixels(0, 0, SCR_WIDTH, SCR_HEIGHT, GL_RGBA, GL_UNSIGNED_BYTE, nullptr);
// now read other PBO which should be already in CPU memory
glBindBuffer(GL_PIXEL_PACK_BUFFER, w_pbo[w_readIndex]);
unsigned char* downsampleData = (unsigned char*)glMapBuffer(GL_PIXEL_PACK_BUFFER, GL_READ_ONLY);

Now you can use unsigned char* downsampleData to build the texture memory