Using NSImageView for video streaming

Question

I am making an voip application which also has video support,

for video, i am getting data as a YUV format and to decode i am using libvpx,

and then i will RGB data,

Now to display i am using NSImageView where i will change the NSImage, please refer the code below

-(void)gotNewRGBBuffer:(void *)imageData Height:(int)height Width:(int)width Scan:(int)scan VideoId:(const char *)pId{

    @autoreleasepool {

        // display the image (on the UI thread)
        NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc]
                                    initWithBitmapDataPlanes:(unsigned char **)&imageData
                                    pixelsWide:width pixelsHigh:height
                                    bitsPerSample:8
                                    samplesPerPixel:3  // or 4 with alpha
                                    hasAlpha:NO
                                    isPlanar:NO
                                    colorSpaceName:NSDeviceRGBColorSpace
                                    bitmapFormat:0
                                    bytesPerRow:scan  // 0 == determine automatically
                                    bitsPerPixel:24];  // 0 == determine automatically


        NSImage *image = [[NSImage alloc] initWithSize:NSMakeSize(width, height)];


        [image addRepresentation:bitmap];

        if ( ![NSThread isMainThread]){

            [self performSelectorOnMainThread:@selector(updateImage:) withObject:image waitUntilDone:NO];
        }else{
            [self updateImage:image];
        }
    }
}

and function to update the image here it goes,

-(void)updateImage:(NSImage *)pNewLocalImage{
            [self pImageView].image = pNewLocalImage;
            [[self pImageView] setNeedsDisplay:YES];
            NSLog(@" Updating the image ");

}

this is working, but taking some considerable amount of memory , so i my question, is there any other way to optimize it!

Answer 1

Thanks for looking at this,

I am now moved to OpenGL NSOpenGLView , which is rendering fast and consuming less cpu and memory,

Refer the code :

GLEssentials

In the given example, it will draw the PNG images , but i need to have YUV buffer, so you need to tweak the vertex and shaders like this,

- (BOOL)setupProgram {
  NSAssert(!_program, @"program already set up");
  GLuint vertexShader = CreateShader(GL_VERTEX_SHADER, kVertexShaderSource);
  NSAssert(vertexShader, @"failed to create vertex shader");
  GLuint fragmentShader =
      CreateShader(GL_FRAGMENT_SHADER, kFragmentShaderSource);
  NSAssert(fragmentShader, @"failed to create fragment shader");
  _program = CreateProgram(vertexShader, fragmentShader);
  // Shaders are created only to generate program.
  if (vertexShader) {
    glDeleteShader(vertexShader);
  }
  if (fragmentShader) {
    glDeleteShader(fragmentShader);
  }
  if (!_program) {
    return NO;
  }
  _position = glGetAttribLocation(_program, "position");
  _texcoord = glGetAttribLocation(_program, "texcoord");
  _ySampler = glGetUniformLocation(_program, "s_textureY");
  _uSampler = glGetUniformLocation(_program, "s_textureU");
  _vSampler = glGetUniformLocation(_program, "s_textureV");
  if (_position < 0 || _texcoord < 0 || _ySampler < 0 || _uSampler < 0 ||
      _vSampler < 0) {
    return NO;
  }
  return YES;
}

Vertex and shader sources,

// Vertex shader doesn't do anything except pass coordinates through.
static const char kVertexShaderSource[] =
  SHADER_VERSION
  VERTEX_SHADER_IN " vec2 position;\n"
  VERTEX_SHADER_IN " vec2 texcoord;\n"
  VERTEX_SHADER_OUT " vec2 v_texcoord;\n"
  "void main() {\n"
  "    gl_Position = vec4(position.x, position.y, 0.0, 1.0);\n"
  "    v_texcoord = texcoord;\n"
  "}\n";

// Fragment shader converts YUV values from input textures into a final RGB
// pixel. The conversion formula is from http://www.fourcc.org/fccyvrgb.php.
static const char kFragmentShaderSource[] =
  SHADER_VERSION
  "precision highp float;"
  FRAGMENT_SHADER_IN " vec2 v_texcoord;\n"
  "uniform lowp sampler2D s_textureY;\n"
  "uniform lowp sampler2D s_textureU;\n"
  "uniform lowp sampler2D s_textureV;\n"
  FRAGMENT_SHADER_OUT
  "void main() {\n"
  "    float y, u, v, r, g, b;\n"
  "    y = " FRAGMENT_SHADER_TEXTURE "(s_textureY, v_texcoord).r;\n"
  "    u = " FRAGMENT_SHADER_TEXTURE "(s_textureU, v_texcoord).r;\n"
  "    v = " FRAGMENT_SHADER_TEXTURE "(s_textureV, v_texcoord).r;\n"
  "    u = u - 0.5;\n"
  "    v = v - 0.5;\n"
  "    r = y + 1.403 * v;\n"
  "    g = y - 0.344 * u - 0.714 * v;\n"
  "    b = y + 1.770 * u;\n"
  "    " FRAGMENT_SHADER_COLOR " = vec4(r, g, b, 1.0);\n"
  "  }\n";