Extract color/depth buffer from OpenGL applications

Question

I was thinking about a project for quiet a while that would require the extraction of the color and depth buffer from OpenGL applications, in particular games. It has absolutely nothing to do with modding, in terms of manipulating the game itself or is intended for "cheating" purposes but more just for data gathering.

So now I'm trying to figure out possible ways to accomplish it. Of course being able to do it with Direct3D under Windows would even lead to more available applications but as I'm pretty familar with OpenGL under Linux, I would start this way.

As there are many modding/cheating applications that actually manipulate the color/depth buffer of video games in different kinds (eg wallhacks in ego shooters), it seems that this definitely has to be possible somehow.

Now the question is, what would be the best way to accomplish this? Reading out the GPU memory directly would most probably not work according to this thread as memory mapping in OpenGL is completely dependent on vendor implementation and there is no trivial way to get the VRAM addresses of the corresponding data.

Alternative approaches I can think of might now be categorized as extravenous or intravenous ones:

• Extravenous : extract the OpenGL context from a process and access buffers, shaders, etc. from a third application without really manipulating the target applications binary directly.
• Intravenous : manipulate the target applications binary/code in such a way, that it writes the correspondings buffers/data either to a specific place in the memory or directly saves it somewhere.

Latter approach should definitely work, but might be associated with a larger effort and would need to be done per application. Hence first would be definitely preferred, but is the described way even applicable at all? Is it just possible to access OpenGL ressources from different processes when you have the OpenGL context value of another? Does anyone has experience with this?

Update:

After some research I found out that what I was looking for is pretty common and called " Hooking " or "Interception" in general.

For OpenGL and Direct3D there are many different libraries and programs to do this:

• glintercept : OpenGL @ Windows
• Indicium-Supra : DirectX
• apitrace : OpenGL + Direct3D @ Windows, macOS, Linux
• D3D9Interceptor : Direct3D
• Nvidia Nsight : Direct3D + OpenGL + Vulkan @ Windows, Linux

and many others.

Answer 1

The de facto standard way of doing this would be to hook yourself into the process and redirect the graphics API calls the application (game) makes to your own code. Your hook can then record whatever data it needs and perform whatever action it wants before passing on the call to the actual API implementation. There are many ways of doing this with different pros and cons, ranging from building a fake library with the same interface and tricking the game into loading that one instead of the actual graphics library ( DLL injection ) to modifying the machine code in the loaded process image to make function calls jump into your code. Which approaches are applicable and best for your case will highly depend on many factors such as target platform, target applications, the API you want to hook into, and so on.

The main issue with any such approach, however, is that that's exactly how many cheats work. Thus, many video games will come with built-in protection to prevent precisely this kind of stuff from working. With many online games, you might even risk having your account suspended for suspected cheating by trying to do stuff like this. But that's just something to be aware of. In my experience, it will still work with many games, particularly single-player games.

"Extracting the OpenGL context from another process" will not work, at least not on any proper operating system. The whole point of having the process abstraction in the first place is to isolate applications from each other…

Since I don't have enough reputation to ask this in a comment, let me ask you here what exactly your goal is. Do you want to get this information once for a single frame or do you want to record this for many frames over a period of time? Do you need an automated solution in form of a custom application? If neither of those, you might be able to just use a graphics debugging tool like Nsight Graphics to capture and export the frame you want…