Too much data on Node.js socket?

Question

I'm currently developing a system that gets data from a battery pack of an electric vehicle, stores it in a database and display it on a screen.

So I have a Java - Application that reads the data from a hardware interface, interprets the values and sends it via Socket to a Node.js-Server. (Java App and Webserver are running on the same computer, so Url = localhost)

JAVA APP:

s = new Socket();
s.connect(new InetSocketAddress(URL, PORT));
out = new PrintWriter( s.getOutputStream(), true);
for (DataEntry e : entries){
        out.printf(e.toJson());
}

NODE:

sock.on('data', function(data) {
    try{
        var data = JSON.parse(data);
        db.serialize(function(){
            db.run("INSERT INTO DataEntry(value, MessageField, time) values(" + data.value + "," + data.messageFieldID + ", STRFTIME('%Y-%m-%d %H:%M:%f'))");
        });
    } catch(e){}
});

I get about 20 Messages per second from the hardware interface which are converted into 100 Json - Strings. So the webserver has to process one message in 10 ms, which I thought, is manageable. But here is the problem: If my entries - List (foreach loop) has more than 2 elements, the webserver gets 2 or more of the Json's in one message.

So the first message was divided into 2 parts (ID 41,42) and was processed correctly. But the second message was divided into 5 parts (ID 43-47), and the first 4 of them weren't sent alone, so only the last one was saved correctly.

How can I ensure, that every Json is sent one another? Isn't there something like a buffer so that the socket.on method is called correctly for every message I send?

I hope somebody of you can help me

Thank you!

Benedikt :)

Answer 1

TCP sockets are just streams and you shouldn't make any assumptions about how much of a "message" is contained in a single packet.

A simple solution to this is to terminate each message with a newline character since JSON cannot contain such a character. From there it's a simple matter of buffering data until you see a newline character. Then call JSON.parse() on your buffer. For example:

var buf = '';
sock.on('data', function(data) {
  buf += data;
  var p;
  // Use a while loop as it may be possible to have multiple
  // messages buffered depending on chunk contents
  while (~(p = buf.indexOf('\n'))) {
    try {
      var msg = JSON.parse(buf.slice(0, p));
    } catch (ex) {
      console.log('Bad JSON message: ' + ex);
    }
    buf = buf.slice(p + 1);
  }
});

You will also need to change printf() to println() on the Java-side so that a newline character will be appended to each message.