Is there a reason why DataInputStream.read() only reads the first few bytes of really big arrays (>100,000 bytes)?

Question

I'm trying to write software that sends a set of data (a portion of a video game) in different formats (chunked, compressed, raw), and measures the speed between each. However, I'm running into an issue while sorting out the CHUNKED method. I've found that, when reading byte array sizes of more than 140000 bytes, the client starts to only read up to around 131072, no matter how much bigger the array actually is. Is there a reason for this, or potentially a better way to do this? My code is shown below. I'm using the read() method of DataInputStream (and its return value).

SERVER

/**
 *
 * @return Time taken to complete transfer.
 */
public int start(String mode, int length) throws IOException, InterruptedException {
    if(mode.equals("RAW")){
        byte[] all = new ByteCollector(ServerMain.FILES, length).collect();
        output.writeUTF("SENDING " + mode + " " + all.length);
        expect("RECEIVING " + mode);
        long start = System.currentTimeMillis();
        echoSend(all);
        return (int) (System.currentTimeMillis() - start);
    }else if(mode.equals("CHUNKED")){ /*the important part*/
        //split into chunks
        byte[] all = new ByteCollector(ServerMain.FILES, length).collect();
        int chunks = maxChunks(all);
        output.writeUTF("SENDING " + mode + " " + chunks);
        System.out.println("Expecting RECEIVING " + chunks + "...");
        expect("RECEIVING " + chunks);
        int ms = 0;
        for(int i = 0; i<chunks; i++){
            byte[] currentChunk = getChunk(i, all);
            System.out.println("My chunk length is " + currentChunk.length);
            long start = System.currentTimeMillis();
            System.out.println("Sending...");
            echoSend(currentChunk);
            ms += System.currentTimeMillis() - start;
        }
        if(chunks == 0) expect("0"); //still need to confirm, even though no data was sent
        return ms;
    }else if(mode.equals("COMPRESSED")){
        byte[] compressed = new ByteCollector(ServerMain.FILES, length).collect();
        compressed = ExperimentUtils.compress(compressed);
        output.writeUTF("SENDING " + mode + " " + compressed.length);
        expect("RECEIVING " + mode);
        long start = System.currentTimeMillis();
        echoSend(compressed, length);
        return (int) (System.currentTimeMillis() - start);
    }
    return -1;
}

public static void main(String[] args) throws IOException,InterruptedException{
    FILES = Files.walk(Paths.get(DIRECTORY)).filter(Files::isRegularFile).toArray(Path[]::new);
    SyncServer server = new SyncServer(new ServerSocket(12222).accept());
    System.out.println("--------[CH UNK ED]--------");
    short[] chunkedSpeeds = new short[FOLDER_SIZE_MB + 1/*for "zero" or origin*/];
    for(int i = 0; i<=FOLDER_SIZE_MB; i++){
        chunkedSpeeds[i] = (short) server.start("CHUNKED", i * MB);
        System.out.println(i + "MB, Chunked: " + chunkedSpeeds[i]);
    }
    short[] compressedSpeeds = new short[FOLDER_SIZE_MB + 1];
    for(int i = 0; i<=FOLDER_SIZE_MB; i++){
        compressedSpeeds[i] = (short) server.start("COMPRESSED", i * MB);
    }
    short[] rawSpeeds = new short[FOLDER_SIZE_MB + 1];
    for(int i = 0; i<=FOLDER_SIZE_MB; i++){
        rawSpeeds[i] = (short) server.start("RAW", i * MB);
    }
    System.out.println("Raw speeds: " + Arrays.toString(rawSpeeds));
    System.out.println("\n\nCompressed speeds: " + Arrays.toString(compressedSpeeds));
    System.out.println("\n\nChunked speeds: " + Arrays.toString(chunkedSpeeds));
}

CLIENT

public static void main(String[] args) throws IOException, InterruptedException {
    Socket socket = new Socket("localhost", 12222);
    DataInputStream input = new DataInputStream(socket.getInputStream());
    DataOutputStream output = new DataOutputStream(socket.getOutputStream());
    while(socket.isConnected()){
        String response = input.readUTF();
        String[] content = response.split(" ");
        if(response.startsWith("SENDING CHUNKED")){
            int chunks = Integer.parseInt(content[2]);
            System.out.println("Read chunk amount of " + chunks);
            output.writeUTF("RECEIVING " + chunks);
            for(int i = 0; i<chunks; i++){
                byte[] chunk = new byte[32 * MB];
                System.out.println("Ready to receive...");
                int read = input.read(chunk);
                System.out.println("Echoing read length of " + read);
                output.writeUTF(String.valueOf(read));
            }
            if(chunks == 0) output.writeUTF("0");
        }else if(response.startsWith("SENDING COMPRESSED")){
            byte[] compressed = new byte[Integer.parseInt(content[2])];
            output.writeUTF("RECEIVING " + compressed.length);
            input.read(compressed);
            decompress(compressed);
            output.writeInt(decompress(compressed).length);
        }else if(response.startsWith("SENDING RAW")){
            int length = Integer.parseInt(content[2]);
            output.writeUTF("RECEIVING " + length);
            byte[] received = new byte[length];
            input.read(received);
            output.writeInt(received.length);
        }
    }
}

public static byte[] decompress(byte[] in) throws IOException {
    try {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        InflaterOutputStream infl = new InflaterOutputStream(out);
        infl.write(in);
        infl.flush();
        infl.close();

        return out.toByteArray();
    } catch (Exception e) {
        System.out.println("Error decompressing byte array with length " + in.length);
        throw e;
    }
}

Using SDK 17

I tried switching around the byte amount, and found the cutoff was right where I stated above. I even replicated this in a test client/server project with no frills (find that here , and found that the cutoff was even lower. I really hope this isn't an actual issue with Java...

Answer 1

The read() method of DataInputStream doesn't directly correspond to the write() method of DataOutputStream. If you want to know how many bytes were sent in a single method call, the server has to inform the client manually.

This is because the read() method, since it doesn't depend on a set length, treats its process as completed when some bytes are read, as it has no way of knowing how many you want.