将Netty直接IO缓冲区累积到CompositeByteBuf

Question

We are migrating from Servlet API 2.5 (thread per request) in our application to Netty. One of cases is using blocking-style ini file parser which we had for many years. Current approach is accumulating incoming ByteBuf into CompositeByteBuf and feeding it to a parser after wrapping with ByteBufInputStream .

In real application we are using HTTP and ini is sent to a server as HTTP request body. But in snippet below it is assumed that all inbound content is transferred file.

class AccumulatingChannelHandler extends ChannelInboundHandlerAdapter {
  final BlockingIniFileParser parser = new BlockingIniFileParser();
  CompositeByteBuf accumulator;

  @Override
  public void channelActive(ChannelHandlerContext ctx) {
    accumulator = ctx.alloc().compositeBuffer(Integer.MAX_VALUE);
  }

  @Override
  public void channelRead(ChannelHandlerContext ctx, Object msg) {
    ByteBuf ioBuffer = (ByteBuf) msg;
    accumulator.addComponent(true, ioBuffer);
  }

  @Override
  public void channelInactive(ChannelHandlerContext ctx) {
    IniFile iniFile = parser.parse(new ByteBufInputStream(accumulator));
    accumulator.release();
    ByteBuf result = process(iniFile);
    ctx.writeAndFlush(result);
    ctx.close();
  }

  private ByteBuf process(IniFile iniFile) {...}
}

class BlockingIniFileParser {
  IniFile parse(InputStream in) {...}
}

interface IniFile {
  String getSetting(String section, String entry);
}

By default pooled direct buffers are coming to channelRead method. And with such strategy we risk to get uncontrollable consumption of direct memory. So, I would like to understand:

1. Is it rational to accumulate IO buffers in such fashion?
2. Is there any best practice for integrating Netty IO with blocking parsers?
3. What is the best practice for parsing input (in some structured format) with Netty?

Answer 1

You have not mentioned the size of a typical uploaded ini file, but assuming they're large enough to cause some concern, I would consider ditching the CompositeByteBuf. Allocate an un-pooled [optionally direct] buffer and as pooled buffers come in the door, write them to the un-pooled buffer, then release them. When you're done reading, continue with your use of a ByteBufInputStream around the un-pooled buffer. Once complete, the un-pooled buffer will be GCed.

You will still be allocating chunks of memory, but it won't be drawn from your buffer pools, and if you use a direct un-pooled buffer, it will not impact your heap as much.

Ultimately, if the size of the un-pooled buffer remains a concern, I would bite the bullet and write the incoming pooled buffers out to disk and on completion, read them back in using a FileInputStream.