Decoding JPEG stream received from IP camera into RGB image in C++

Question

I have an IP camera which streams JPEG compressed data in UDP/RTP. I used Boost Asio to receive the data. I know I have received the data correctly because I checked the header information for each packet with Wireshark.

#include <boost/asio.hpp>

#define MAX_BYTES 1500 

using boost::asio::ip::udp;

int main(){

  boost::asio::io_service io_service;
  udp::socket socket( io_service , udp::endpoint(udp::v4(),50242) );
  char data[MAX_BYTES];
  size_t bytes_received = 0;

  while(true){

    bytes_received = socket.receive(boost::asio::buffer(data,MAX_BYTES));

  }

}

I receive the total of 1452 bytes for each packet, the first 20 bytes of which are RTP header(12 bytes) followed by JPEG header(8 bytes). The remaining 1432 bytes contain the payload. Let's say, each frame is made of 145 packets (frm_pckts = 145) . After sorting the packets I store them in a buffer for a single frame as follows:

unsigned char* buffer = (unsigned char*) malloc( frm_pckts * 1432 * sizeof(unsigned char);  

memcpy( &buffer[packet_index*1432] , &data[20], 1432);

if I copy this buffer into an OpenCV Mat and display that it will display garbage values. I also tried

cv::Mat img = cv::imdecode(frame, CV_LOAD_IMAGE_COLOR);

but it didn't work either. I looked all the possible solutions I could find on this website but none of them worked for me. Is there any library out there that I could use to pass this buffer to one of it's functions and retrieve the JPEG frame? Also, for decoding the frame should I include the header bytes in the buffer as well? If yes which header? JPEG header, RTP header or both?

I would very much appreciate any useful solutions or suggestions.

Answer 1

The JPEG over RTP header is called the main header and is only 8 bytes long. from that the JFIF header should be created in order to be able to decode the JPEG compressed data. The JFIF header creation can be done using "RFC 2435" format. Everything has been explained there. Once the JFIF header is created, it should be prepended to the compressed data and then passed to uncompressedFrame = cv::imdecode(compressedImage, 1) or any other JPEG decoders to retrieve the uncompressed frame. If you run into any problems like this, please let me know at (davar.daei@gmail.com) I will be more than happy to help.