Android虚拟显示的帧侦听器（NDK内部版本）

Question

I'm building an internal shared library for Android platform. I have the signing keystore from the device manufacturer.

My library is making use of ScreenRecord.cpp internal file from Android source. Recording works fine with the MediaCodec encoder; however I want to access each frame so that I can apply some image overlay logo on to each frame before it gets passed to the encoder. There's an overlay example in Android source too, but that only works for newer versions of Android (5.0 / API 21+). I want to have an overlay solution for Android Kitkat (4.4 / API 19)

Here's a code example that I obtained from minicap .

mVirtualDisplay = android::SurfaceComposerClient::createDisplay(
android::String8("minicap"),
true);

LOGI("Creating buffer queue");
mScreenshotClient.getCpuConsumer();
mBufferQueue = mScreenshotClient.mBufferQueue;

LOGI("Creating CPU consumer");
mConsumer = new android::CpuConsumer(mBufferQueue, 3, false);
mConsumer->setName(android::String8("minicap"));
mConsumer->setDefaultBufferSize(targetWidth, targetHeight);
mConsumer->setDefaultBufferFormat(android::PIXEL_FORMAT_RGBA_8888);

mConsumer->setFrameAvailableListener(mFrameProxy); 
//mFrameProxy is from:
//class FrameProxy: public android::ConsumerBase::FrameAvailableListener

LOGI("Publishing virtual display");
android::SurfaceComposerClient::openGlobalTransaction();
android::SurfaceComposerClient::setDisplaySurface(mVirtualDisplay, mBufferQueue);
android::SurfaceComposerClient::setDisplayProjection(mVirtualDisplay,
android::DISPLAY_ORIENTATION_0, layerStackRect, visibleRect);
android::SurfaceComposerClient::setDisplayLayerStack(mVirtualDisplay, 0);// default stack

android::SurfaceComposerClient::closeGlobalTransaction();

I set up the above code, but onFrameAvailable() method of FrameAvailableListener gets called only once. It never gets called again even when I do stuff on the screen. What am I missing here? Isn't there any lesser tricky way to access the frames before passing to the encoder?

Answer 1

An example of adding an overlay is built into the screenrecord sources for Lollipop. As far as I can recall it doesn't rely on any features added in Lollipop, so you should be able to build and run it on 4.4. As noted on bigflake , the --bugreport mode was added to AOSP in the 4.4 time frame, but didn't actually ship with the system until 5.x. (With a minor tweak, it should even run on 4.3, but I haven't tried it.)

The key source files are Overlay.{cpp,h} . It does the same thing that you would do from code written in Java: create a GLConsumer (SurfaceTexture), use that to convert the incoming frames to GLES textures, then render texture + overlay to the video encoder.

Sample video is here . Note it adds a block of text to the very start, and a running timestamp / frame-counter to the top left corner.

Note for anyone else reading this: this code is using internal private APIs that have been changing in recent releases, so any binaries must be built for specific versions of Android, and may not be portable to devices built by different manufacturers even if they run the same version of Android (sometimes the OEMs like to mess with things).

Update: My earlier statements about working on KitKat weren't accurate -- there was a major API shift before the Lollipop version went out. The trick is to grab the sources before this change went in, as that was when the BufferQueue API rewrite reached screenrecord. You can see from the change list that the --bugreport option went in about five months before that.