Netty: MessageToMessageEncoder and conversion to ByteBuf

Question

I'm new to Netty and trying to understand how is working.

I want to create a simple SMTP server for testing purposes, on localhost and I'm having a small problem understanding how encoder works.

First here is what I want to achieve.

1. Client is connecting to server on port 25 - OK
2. Server sends this message: 220 localhost Test \\r\\n
3. Client receives the message sent at 2 and reply with HELO whatever

My problem is at point 2

I'm trying to use MessageToMessageEncoder

@Override
protected void encode(ChannelHandlerContext ctx, Object msg, List<Object> out) throws Exception {
    //...
    out.add("220 localhost test \r\n");
}

That code out.add("220 localhost Test \\r\\n"); is doing nothing, no error is returned and the client is not proceeding to point 3 sending the HELO message.

However if I change that to this

ByteBuf buff = ByteBufUtil.writeUtf8(ctx.alloc(), "220 localhost Test \r\n");
out.add(buff);

Everything is running great and the client is sending the HELO message

So my question is do I always need to encode the message to ByteBuf ? I'm asking this because on official documentation: http://netty.io/4.1/api/io/netty/handler/codec/MessageToMessageEncoder.html I see something like this out.add(message.toString());

Answer 1

Yes. Eventually, the payload must be in the form of a ByteBuf. However, as intended, but not displayed in the example you cited, the MessageToMessage encoder would be followed in the downstream pipeline by another encoder that converts the message into a ByteBuf. In your case, a http://netty.io/4.1/api/io/netty/handler/codec/string/StringEncoder.html would do the trick, as seen in this code from that link:

ChannelPipeline pipeline = ...;

 // Decoders
 pipeline.addLast("frameDecoder", new LineBasedFrameDecoder(80));
 pipeline.addLast("stringDecoder", new StringDecoder(CharsetUtil.UTF_8));

 // Encoder
 pipeline.addLast("stringEncoder", new StringEncoder(CharsetUtil.UTF_8));