FFMPEG和JNI - 将AVFrame数据传递给Java和Back

Question

I have some C code that decodes a video frame by frame. I get to a point where i have an AVFrame in BGR32 and would like to send it back to Java for editing.

I have a ByteBuffer object in my C code that was created in Java using AllocateDirect but i struggle to write the content of the AVFrame->data[0] (of uint8_t type) to it and read it back. I have tried memcpy with no luck. Does anyone have an idea how to achieve this?

UPDATE Followed Will's comment below and wrote this in C

char *buf = (*pEnv)->GetDirectBufferAddress(pEnv, byteBuffer);
memcpy(buf, rgb_frame->data[0], output_width*output_height*4);

The buffer does contain some data in Java but doing the following returns a null bitmap

BufferedImage frame = ImageIO.read(bitmapStream);

Where bitmapStream is a ByteBufferInputStream defined here: https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/ByteBufferInputStream.java?r=205

Not sure if I am not writing things correctly in this buffer

UPDATE 2

Got pretty close now thanks to the latest snippet. I am using BGR32 format in my C code ie 4 bytes per pixel. So I modified things a bit in Java:

final byte[] dataArray = new byte[width*height*4];
imageData.get(dataArray);
final BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_BGR);
final DataBuffer buffer = new DataBufferByte(dataArray, dataArray.length);
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);

I get the image correctly but there seems to be an issue with color channels

Tried different formats with no luck

Answer 1

From Oracle's JNI Functions Documentation at
https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#GetDirectBufferAddress

GetDirectBufferAddress

void* GetDirectBufferAddress(JNIEnv* env, jobject buf);

Fetches and returns the starting address of the memory region referenced by the given direct java.nio.Buffer.

This function allows native code to access the same memory region that is accessible to Java code via the buffer object. LINKAGE:

Index 230 in the JNIEnv interface function table. PARAMETERS:

env: the JNIEnv interface pointer

buf: a direct java.nio.Buffer object (must not be NULL) RETURNS:

Returns the starting address of the memory region referenced by the buffer. Returns NULL if the memory region is undefined, if the given object is not a direct java.nio.Buffer, or if JNI access to direct buffers is not supported by this virtual machine. SINCE:

JDK/JRE 1.4

I tested with this C++ code:

static char framebuf[100];

JNIEXPORT void JNICALL Java_javaapplication45_UseByteBuffer_readBuf
  (JNIEnv *env, jobject usebb, jobject bb) {
    void *addr = env->GetDirectBufferAddress(bb);
    framebuf[0] = 77;
    memcpy(addr,framebuf,100);
}

and this Java Code:

public class UseByteBuffer {
    public native void readBuf(ByteBuffer bb);
}

...

public static void main(String[] args) {
    System.load("/home/shackle/NetBeansProjects/usebb/dist/Debug/GNU-Linux-x86/libusebb.so");
    ByteBuffer bb = ByteBuffer.allocateDirect(100);
    new UseByteBuffer().readBuf(bb);
    byte first_byte = bb.get(0);
    System.out.println("first_byte = " + first_byte);
}

And it printed the first_byte=77 indicating it got the data copied correctly.

Update

ImageIO.read() will not accept just any set of bytes it has to be in a format that one of the installed ImageReader's can recognize such as JPEG or PNG.

Here is an example getting raw for (3 byte r,g,b )bytes into an image

int width = 256;
int height = 256;
ByteBuffer bb = ByteBuffer.allocateDirect(height*width*3);

byte[] raw = new byte[width * height * 3];
bb.get(raw);
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_3BYTE_BGR);
DataBuffer buffer = new DataBufferByte(raw, raw.length);
SampleModel sampleModel = new ComponentSampleModel(DataBuffer.TYPE_BYTE, width, height, 3, width * 3, new int[]{0,1,2});
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);

Update 2

For BGR32 I believe this would be closer:

ByteBuffer imageData = ByteBuffer.allocateDirect(height * width * 4);
byte[] raw = new byte[width * height * 4];
imageData.get(raw);
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_BGR);
DataBuffer buffer = new DataBufferByte(raw, raw.length);
SampleModel sampleModel = new ComponentSampleModel(
        DataBuffer.TYPE_BYTE, width, height, 4, width * 4,
        new int[]{2,1,0} // Try {1,2,3}, {3,2,1}, {0,1,2}
);
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);

Notice where I have commented, where I suspect you may need to experiment with the array of bandOffsets in the third argument of the ComponentSampleModel constructor to fix the color model.

Update 3

One can reuse the sampleModel to get data out of the image by using BufferedImage.copyData() to a WritableRaster instead of using getRaster().

SampleModel sampleModel = new ComponentSampleModel(
        DataBuffer.TYPE_BYTE, width, height, 4, width * 4,
        new int[]{2, 1, 0} 
);

...

BufferedImage newImage = ImageIO.read(new File("test.png"));
byte newRaw[] = new byte[height*width*4];
DataBuffer newBuffer = new DataBufferByte(newRaw, newRaw.length);
WritableRaster newRaster = Raster.createWritableRaster(sampleModel, newBuffer, null);
newImage.copyData(newRaster);