Avro Communication over TCP Sockets

Question

I am currently working on a project that will involve communication of applications written in C and Java. Therefore, I chose to work with Apache Avro. I have seen on the website that Avro can (de-)serialize objects from files using the DataFileWriter class.

But, in my case I want to use TCP sockets between my applications. Therefore, DataFileWriter class is not going to work for me. After reading the documentation, I have not found any information on how to send objects through TCP sockets.

Any ideas on how to do that? I specifically want to know what kind of Input and Output Streams I should use on the Java Clients.

I have developed the following code for the Java Server:

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
import java.util.HashMap;

import middleman.bigpeer.BigPeer;

import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.BinaryEncoder;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;

public class MiddleManWorker implements Runnable {

    private InputStream in;

    private OutputStream out;

    private Socket clientSocket;

    public MiddleManWorker(Socket clientSocket, HashMap<Integer, NodeType> dbNodesDirectory, 
            HashMap<Integer, NodeType>  workersDirectory) {
        this.clientSocket = clientSocket;
        try {
            this.in = clientSocket.getInputStream();
            this.out = clientSocket.getOutputStream();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Override
    public void run() {
        EncoderFactory encoderFactory = new EncoderFactory();
        DecoderFactory decoderFactory = new DecoderFactory();
    BinaryEncoder binaryEncoder = encoderFactory.binaryEncoder(out, null);
        BinaryDecoder binaryDecoder = decoderFactory.binaryDecoder(in, null);
        SpecificDatumReader<BigPeer> peerDatumReader = new SpecificDatumReader<BigPeer>(BigPeer.class);
        BigPeer bigPeer = null;
        SpecificDatumWriter<BigPeer> writer = new SpecificDatumWriter<BigPeer>();
        try {
            peerDatumReader.read(bigPeer, binaryDecoder);
            System.out.println("Received: " + bigPeer.getType());
        } catch (IOException e) {
            e.printStackTrace();
        }
        try {
            writer.write(bigPeer, binaryEncoder);
        } catch (IOException e) {
            e.printStackTrace();
        }


    }

} 

A sample Java Client is the following:

import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.net.Socket;

import middleman.bigpeer.BigPeer;

import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.BinaryEncoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;

public class SystemClient {

    public static void connect(String serverIPAddress, Integer serverPort) throws IOException, ClassNotFoundException {
        /**
         * Create Connection with the server
         */
        Socket socket = new Socket(serverIPAddress, serverPort);
        InputStream in = socket.getInputStream();
        OutputStream out = socket.getOutputStream();
        EncoderFactory encoderFactory = new EncoderFactory();
        DecoderFactory decoderFactory = new DecoderFactory();
        BinaryEncoder binaryEncoder = encoderFactory.binaryEncoder(out, null);
        BinaryDecoder binaryDecoder = decoderFactory.binaryDecoder(in, null);

        BigPeer bigPeer = new BigPeer();
        bigPeer.setType("test");
        SpecificDatumReader<BigPeer> reader = new SpecificDatumReader<BigPeer>(BigPeer.class);
        SpecificDatumWriter<BigPeer> writer = new SpecificDatumWriter<BigPeer>(BigPeer.class);

        System.out.println("Before: " +  bigPeer.getType());
        writer.write(bigPeer, binaryEncoder);
        System.out.println("Waiting for response...");
        reader.read(bigPeer, binaryDecoder);
        System.out.println("After: " + bigPeer.getType());
    }

}

And the server seems to halt on the peerDatumReader.read(bigPeer, binaryDecoder); line of code. Any ideas?

Thank you, Nick

Answer 1

BinaryEncoder uses an internal buffer for performance reasons. You may need to call flush on the encoder to send the data through the pipe.

See the reference for more information on this behaviour:

The BinaryEncoder implementation returned may buffer its output. Data may not appear on the underlying OutputStream until Flushable.flush() is called. The buffer size is configured with configureBufferSize(int).

If buffering is not desired, and lower performance is acceptable, use directBinaryEncoder(OutputStream, BinaryEncoder)