libx264 x264_nal_encode overflowing

Question

My libH264 app is performing encoding of incomming camera data, but it keeps accumulating data and slows down the more encode operations it performs. Any idea's why is this happening?

Variables are allocated at the start of a program. Is my sequence incorrect? I suspect I have to reinitilize some structure, but I cant find out how to do it...

Here is part of my code in camera thread:

    int nWidth = 640;
    int nHeight = 480;    

fill YUV data

    memcpy(p264Pic->img.plane[0],pic.buffer,nWidth*nHeight);  
    memcpy(p264Pic->img.plane[1],pic.buffer+nWidth*nHeight,nWidth*nHeight/4);  
    memcpy(p264Pic->img.plane[2],pic.buffer+nWidth*nHeight*5/4,nWidth*nHeight/4); 

fill PTS

    p264Pic->i_pts = high_resolution_timer_tick_count();

//          if( nFramsInPack % 8 == 0){
//           p264Pic->i_type = X264_TYPE_I;
//          }else{
             p264Pic->i_type = X264_TYPE_AUTO;
//      }

    if( x264_encoder_encode( p264Handle, &p264Nal, &i264Nal, p264Pic ,&pic_out) < 0 ) {  
                fprintf( stderr, "x264_encoder_encode failed/n" );  
    }  
    printf("i264Nal %d\n", i264Nal );

    for( int i = 0; i < i264Nal; i++ )  
    {
     printf( "nal pass %d\n", i );
     int i_data = 1024*32;  

     printf( "nal encode\n" );

     x264_nal_encode( p264Handle, pNal, &p264Nal[i] );

     printf( "after nal encode\n" );

     int i_size = p264Nal[i].i_payload;     

     printf("i_size %d\n", i_size );

     if( i_size > 0 ) {  

       if ((pNal[4]&0x60)==0) {  
           continue;  
       }  
       if (pNal[4]==0x67) {  
           continue;  
       }  
       if (pNal[4]==0x68) {  
           continue;  
       }  
       memmove(pNal,pNal+4,i_size-4);  
       pNal+=i_size-4;  
     }  
     else if( i_size < 0 ) {  
       fprintf( stderr,"need to increase buffer size (size=%d)/n", -i_size );  
     }  
    }  

Pack data to RTMP

    unsigned int nSize=pNal-szNalBuffer;  
    packet.m_nBodySize=nSize+9;  
    if (i264Nal>1) {  
     szBodyBuffer[ 0]=0x17;  
    }  
    else {  
     szBodyBuffer[ 0]=0x27;  
    }  
    put_be32((char *)szBodyBuffer+5,nSize);  
    memcpy(szBodyBuffer+9,szNalBuffer,pNal-szNalBuffer);  

    printf( "rtmp send packet\n" );

    RTMP_SendPacket(rtmp,&packet,0);  
    newTick=high_resolution_timer_tick_count();  
    packet.m_headerType = RTMP_PACKET_SIZE_MEDIUM;  
    packet.m_nTimeStamp+=newTick-oldTick;  
    oldTick=newTick;  

Here is the output of my app:

i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 0
rtmp send packet
i264Nal 4
nal pass 0
nal encode
after nal encode
i_size 36
nal pass 1
nal encode
after nal encode
i_size 15
nal pass 2
nal encode
after nal encode
i_size 632
nal pass 3
nal encode
after nal encode
i_size 7151
rtmp send packet
i264Nal 1
nal pass 0
nal encode
after nal encode
i_size 8136
rtmp send packet
i264Nal 1
nal pass 0
nal encode
after nal encode
i_size 28955
rtmp send packet
^CThe pthread schedule_do end
^Cthread is end
i264Nal 1
nal pass 0
nal encode
after nal encode

Answer 1

x264_encoder_encode(...) returns the frame size, you don't seem to be taking this into consideration . Your using i_payload but depending on the NAL type, this will not be the correct value.

x264_nal_encode looks like it was removed and x264_encoder_encode now returns the frame encoded into nal units. int frame_size is returned.