How to send compressed data using UDP or TCP (lua / java)

Question

I am making a server with lua clients and Java server. I need some data to be compressed in order to reduce the data flow.

In order to do this I use LibDeflate for compressing the data on the client side

local config = {level = 1}
local compressed = LibDeflate:CompressDeflate(data, config)
UDP.send("21107"..compressed..serverVehicleID) -- Send data

On the server I use this to receive the packet (TCP)

out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedReader(new 
InputStreamReader(clientSocket.getInputStream(), "UTF-8"));
String inputLine;

while ((inputLine = in.readLine()) != null) { // Wait for data
    Log.debug(inputLine); // It is what get printed in the exemple
    String[] processedInput = processInput(inputLine);
    onDataReceived(processedInput);
}

I already tried sending it using UDP and TCP, the problem is the same. I tried using LibDeflate:CompressDeflate and LibDeflate:CompressZlib I tried tweaking the config Nothing works :/

I expect to receive one packet with the whole string But I received few packets each of them contains compressed characters. exemple (each line is the server think that he receive a new packet):
_{(source: noelshack.com )}

Answer 1

After a lot of research I finnaly managed to fix my problem ! I used this :

DataInputStream in = new DataInputStream(new BufferedInputStream(clientSocket.getInputStream()));

int count;
byte[] buffer = new byte[8192]; // or 4096, or more

while ((count = in.read(buffer)) > 0) {
    String data = new String(buffer, 0, count);
    Do something...
}

I still haven't tested to see if the received compressed string works, I'll update my post when I try out.

EDIT: It seems to work

The only problem now is that I don't know what to do when the packet is bigger than the buffer size. I want to have something that work in every situation and since some packet are bigger than 8192 they are just cut in half.

Answer 2

Assuming that the client side sends a single compressed "document", your server-side code should look something like this (TCP version):

is = new DeflaterInputStream(clientSocket.getInputStream());
in = new BufferedReader(new InputStreamReader(is, "UTF-8"));
String inputLine;

while ((inputLine = in.readLine()) != null) { 
    ...
}

The above is untested, and also needs exception handling and code to ensure that the streams always get closed.

The trick is that your input pipeline needs to decompress the data stream before you attempt to read / process it as lines of text.