Using FFMPEG to make HLS clips from H264

Question

I am using a Hi35xx camera processor from HiSilicon. It is an Arm9 with a video pipeline bolted on the side. At one end of the pipeline is the CMOS sensor. At the other end is a H264 encoder. When I turn on the pipeline, the encoder outputs H264 NAL packets like this:

frame0: <SPS>,<PPS>,<SEI>,<key frame>
frame1: <delta frame>
frame2: <delta frame>
...
frameN: <delta frame>
frameN+1: <SPS>,<PPS>,<SEI><key frame>
frameN+2: <delta frame>
frameN+3: <delta frame>
...
etc.

I am turning that into HLS clips by doing the following (pseudo code for clarity) :

av_register_all();
avformat_network_init();

avformat_alloc_output_context2(&ctx_out, NULL, "hls", "./foo.m3u8");

strm_out = avformat_new_stream(ctx_out, NULL);

codec_out = strm_out->codecpar;
codec_out->codec_id = AV_CODEC_ID_H264;
codec_out->codec_type = AVMEDIA_TYPE_VIDEO;
codec_out->width = encoder_width;
codec_out->height = encoder_height;
codec_out->bit_rate = encoder_bitrate;
codec_out->codec_tag = 0;

avformat_write_header(ctx_out, NULL);

while(get_packet_from_pipeline_encoder(&encoder_packet)) {
  AVPacket pkt;
  av_init_packet(&pkt);
  pkt.stream_index = 0;

  pkt.dts = AV_NOPTS_VALUE;
  pkt.pts = AV_NOPTS_VALUE;
  pkt.duration = (1000000/FRAMERATE);    // frame rate in microseconds

  pkt.data = encoder_packet.data;
  pkt.size = encoder_packet.size;

  if (is_keyframe(&encoder_packet)) {
    pkt.flags |= AV_PKT_FLAG_KEY;
  }

  av_write_frame(ctx_out, &pkt);
}

av_write_trailer(ctx_out);
avformat_free_context(ctx_out);

This seems to work fine except that the resulting HLS frame rate is not right. Of course, this happens because I am not setting the pts/dts stuff correctly and ffmpeg lets me know that. So I have two quetions:

1. Am I going about this right?
2. How can I set the pts/dts stuff correctly?

The encoder is giving me packets and I am submitting them as frames. Those <SPS>, <PPS> and <SEI> packets are really out of band data and don't really have a timestamp. How can I submit them correctly?

Answer 1

My conclusion is that I was going at this the wrong way. The basic problem was that I don't have an input context so there is no h264 parser that can take the SPS, PPS and SEI packets and do anything with them. I suspect that my loop appears to be working because I am writing to an 'mpegts' file which is just h264 packets with the leading NAL zeroes replace with length words (some bit-stream filter is doing that). But that would mean that there isn't much chance that I can get the timestamps right because I have to submit them as frames. I can't submit them as 'extradata/sidedata' because there is no decoder to catch them.

This problem is fixed by writing a custom IO context for the output of my encoder and then do a normal input context. I have done some experiments with this approach and it seems to work.