fully real-time screen capture in window 8/10 technology without delay

Question

I'm gonna create a very fast and real-time remote service with C# or C++ , doesn't matter , but currently I'm using c# .

okay , What I'm looking for and asking for is a way to capture windows screen at real time in fastest shape .

I know the ways already is on internet but the problem is the delay ...

• I used C# copyfromscreen and the delay was 92ms for 1920x1080 [ 5 frame in video ]
• I used C# copyfromscreen with jpg encoder and delay was 36ms for 1920x1080
• I used Unity3D screenCapture with jpg encoder and delay was 38ms for 1920x1080
• I used C# Windows Desktop Duplication API and delay was 19ms for 1920x1080 [ 3 frame in video ]
• I used C# Windows Desktop Duplication API with jpg encoder and delay was 12ms for 1920x1080 [ 2 frame in video ]
• I used C++ Windows Desktop Duplication API with jpg encoder and delay was 9ms for 1920x1080 [ 2 frame in video ]
• I used C# Windows DirectXCapture with jpg encoder and delay was 16ms for 1920x1080 [ 2 frame in video ]

I thought the progress is normal at this way till I checked out Windows 10/8 Task Bar Live Preview Thumbnails and It's fully-real time with out 1ms delay that means frame to frame !

I tried to resize 1920 x 1080 from all methods to Task Bar Preview size but it didn't made any change !

Note : This program is not processing over internet it's gonna work on local network.

I'm guessing the delay is because of bitmap processing or something but I don't know yet! [Times Added] Here's the way I process picture on server :

private static byte[] capture()
{
    Bitmap bmp = screenshot_DDAPI(); // Avarage Time : 29.6566ms
    MemoryStream ms = new MemoryStream();
    bmp.Save(ms, ImageFormat.Jpeg); // Avarage Time : 1.7101 ms
    return ms.ToArray(); // Avarage Time : 3.032 ms
}

Anybody knows which technology and method windows uses to process a screenshot to task bar live thumbnails ?

Note : if this question is not related to stack-overflow or it's off-topic please tell me which stack category I can ask my question ?

Thanks

Answer 1

Desktop duplication API is the fastest one. The capture delay is zero.

However, after the capture, you're downloading raw texture data into system RAM for CPU-based jpeg encoder. This is what taking the time. Live thumbnails don't need to do that, they scale window texture in GPU and render it with GPU, too, both are very cheap when the source data is already in VRAM.

If you really want to minimize latency, look for GPU-based JPEG encoder that can take source data from B8G8R8A8_UNORM D3D11 texture. JPEGs take much less space (and therefore bandwidth) than RGBA, ie you'll probably get the encoded result sooner.

Answer 2

Probably the fastest way to do this is to use DirectX. Every DirectX application contains a buffer, or a surface to hold the contents of the video memory related to that application. This is called the back buffer of the application. Some applications might have more than one back buffer. And there is another buffer that every application can access by default - the front buffer. This one, the front buffer, holds the video memory related to the desktop contents, and so essentially is the screen image. By accessing the front buffer from your application, you can capture the contents of the screen at that moment.

Accessing the front buffer from your application is pretty easy and straightforward. The interface IDirect3DDevice9 provides the GetFrontBufferData() method that takes a IDirect3DSurface9 object pointer and copies the contents of the front buffer onto that surface. The IDirect3DSurfce9 object can be generated by using the method IDirect3DDevice8::CreateOffscreenPlainSurface() . Once the screen is captured onto the surface, you can use the function D3DXSaveSurfaceToFile() to save the surface directly to the disk in bitmap format. Thus, the code to capture the screen looks as follows:

extern IDirect3DDevice9* g_pd3dDevice;
Void CaptureScreen()
{
    IDirect3DSurface9* pSurface;

    // g_pd3dDevice is an IDirect3DDevice9 object, and has been assumed to be properly initialized
    g_pd3dDevice->CreateOffscreenPlainSurface(ScreenWidth, ScreenHeight,
        D3DFMT_A8R8G8B8, D3DPOOL_SCRATCH, &pSurface, NULL);
    g_pd3dDevice->GetFrontBufferData(0, pSurface);
    D3DXSaveSurfaceToFile("Desktop.bmp",D3DXIFF_BMP,pSurface,NULL,NULL);
    pSurface->Release(); 
}

If you want a buffer to the actual bits, instead of saving it directly to disk, you can use code like this:

extern void* pBits;
extern IDirect3DDevice9* g_pd3dDevice;
IDirect3DSurface9* pSurface;

// g_pd3dDevice is an IDirect3DDevice9 object, and has been assumed to be properly initialized
g_pd3dDevice->CreateOffscreenPlainSurface(ScreenWidth, ScreenHeight,
                                          D3DFMT_A8R8G8B8, D3DPOOL_SCRATCH, 
                                          &pSurface, NULL);
g_pd3dDevice->GetFrontBufferData(0, pSurface);
D3DLOCKED_RECT lockedRect;
pSurface->LockRect(&lockedRect,NULL,
                   D3DLOCK_NO_DIRTY_UPDATE|
                   D3DLOCK_NOSYSLOCK|D3DLOCK_READONLY)));
for( int i=0 ; i < ScreenHeight ; i++)
{
    memcpy( (BYTE*) pBits + i * ScreenWidth * BITSPERPIXEL / 8 , 
        (BYTE*) lockedRect.pBits + i* lockedRect.Pitch , 
        ScreenWidth * BITSPERPIXEL / 8);
}
g_pSurface->UnlockRect();
pSurface->Release();

Make sure that we have allocated enough memory before copying into pBits . A typical value for BITSPERPIXEL is 32 bits per pixel. However, it may vary depending on your current monitor settings.

EDIT

Here is a C++ program I put together to time it. The function Direct3D9TakeScreenshots takes a parameter that determines how many times to take a screen shot. I have it set to 10 right now, but you can change it. Running on my machine, in release mode, I get the following results:

18:33:23.189
18:33:23.579

So that's 390 milliseconds to take 10 screen shots and copy them to a buffer. That averages to about 39 milliseconds per screen shot and buffer copy.

#include "pch.h"
#include <iostream>

#include <d3d9.h>                 // DirectX 9 header
#pragma comment(lib, "d3d9.lib")  // link to DirectX 9 library

#define WIDEN2(x) L ## x
#define WIDEN(x) WIDEN2(x)
#define __WFILE__ WIDEN(__FILE__)
#define HRCHECK(__expr) {hr=(__expr);if(FAILED(hr)){wprintf(L"FAILURE 0x%08X (%i)\n\tline: %u file: '%s'\n\texpr: '" WIDEN(#__expr) L"'\n",hr, hr, __LINE__,__WFILE__);goto cleanup;}}
#define RELEASE(__p) {if(__p!=nullptr){__p->Release();__p=nullptr;}}

HRESULT Direct3D9TakeScreenshots(UINT adapter, UINT count)
{
    HRESULT hr = S_OK;
    IDirect3D9 *d3d = nullptr;
    IDirect3DDevice9 *device = nullptr;
    IDirect3DSurface9 *surface = nullptr;
    D3DPRESENT_PARAMETERS parameters = { 0 };
    D3DDISPLAYMODE mode;
    D3DLOCKED_RECT rc;
    UINT pitch;
    SYSTEMTIME st;
    LPBYTE *shots = nullptr;

    // init D3D and get screen size
    d3d = Direct3DCreate9(D3D_SDK_VERSION);
    HRCHECK(d3d->GetAdapterDisplayMode(adapter, &mode));

    parameters.Windowed = TRUE;
    parameters.BackBufferCount = 1;
    parameters.BackBufferHeight = mode.Height;
    parameters.BackBufferWidth = mode.Width;
    parameters.SwapEffect = D3DSWAPEFFECT_DISCARD;
    parameters.hDeviceWindow = nullptr;

    // create device & capture surface
    HRCHECK(d3d->CreateDevice(adapter, D3DDEVTYPE_HAL, NULL, D3DCREATE_SOFTWARE_VERTEXPROCESSING, &parameters, &device));
    HRCHECK(device->CreateOffscreenPlainSurface(mode.Width, mode.Height, D3DFMT_A8R8G8B8, D3DPOOL_SYSTEMMEM, &surface, nullptr));

    // compute the required buffer size
    HRCHECK(surface->LockRect(&rc, NULL, 0));
    pitch = rc.Pitch;
    HRCHECK(surface->UnlockRect());

    // allocate screenshots buffers
    shots = new LPBYTE[count];
    for (UINT i = 0; i < count; i++)
    {
        shots[i] = new BYTE[pitch * mode.Height];
    }

    GetSystemTime(&st); // measure the time we spend doing <count> captures
    wprintf(L"%i:%i:%i.%i\n", st.wHour, st.wMinute, st.wSecond, st.wMilliseconds);
    for (UINT i = 0; i < count; i++)
    {
        // get the data
        HRCHECK(device->GetFrontBufferData(0, surface));

        // copy it into our buffers
        HRCHECK(surface->LockRect(&rc, NULL, 0));
        CopyMemory(shots[i], rc.pBits, rc.Pitch * mode.Height);
        HRCHECK(surface->UnlockRect());
    }
    GetSystemTime(&st);
    wprintf(L"%i:%i:%i.%i\n", st.wHour, st.wMinute, st.wSecond, st.wMilliseconds);


cleanup:
    if (shots != nullptr)
    {
        for (UINT i = 0; i < count; i++)
        {
            delete shots[i];
        }
        delete[] shots;
    }
    RELEASE(surface);
    RELEASE(device);
    RELEASE(d3d);
    return hr;
}

int main()
{
    Direct3D9TakeScreenshots(D3DADAPTER_DEFAULT, 10);
}