删除分配给字节数组的 memory

Question

I am receiving records in while loop at a socket server. where each records has a msg type followed by message length and actual message content.

The problem is that since I get around a million records and each record has a record size of 277 bytes. So, after around 40000 records, I get OutOfMemory error. The code flow looks something like this:

while (true) {              
            msgType = dIn.readByte();

            int msgIntType = msgType & 0xff;

                  // get message length

                  int msgIntLen = dIn.readInt();
                  if (msgIntLen != 0) {

     msgContent = new byte[msgIntLen];
                   switch(msgIntType) {
            case 4:
            //case MSG_DATA:
                // MSG_DATA
                recordCount++;
                processData(msgContent);
                if (recordCount == 2000) {
                sendACK(dOut, msgIntType);
                logger.info("sent ACK for MSG_DATA");
                recordCount = 0;
                }               
                break;

}

I resolved the OutOfMemory issue by explicitly calling System.gc() after sending ACK after every 2000 records processed and now it works perfectly fine and able to handle 1 million records without any error in less than 10 minutes. The modified code for case statement for calling System.gc() looks like:

            case 4:
            //case MSG_DATA:
                // MSG_DATA
                recordCount++;
                processData(msgContent);
                if (recordCount == 2000) {
                sendACK(dOut, msgIntType);
                logger.info("sent ACK for MSG_DATA");
                recordCount = 0;
                             System.gc();
                }               
                break;

But I read here on some other posts that calling System.gc() is not a good design approach? Is it so? If yes, could you guys suggest me some other way to get rid of this OutOfMemory error?

Thanks in advance -JJ

EDIT: logic for processData():

public void processData(byte[] msgContent) throws Exception {

    InputStreamReader inp = new InputStreamReader(

            new ByteArrayInputStream(msgContent));

    BufferedReader br = null;
    try {

        br = new BufferedReader(inp);
                             String line;
        while ((line = br.readLine()) != null) {

                             process each line
                             .
                             }
                  } catch (Exception e) {
        logger.error("exception in " + Utils.getExecutingMethodName(e) 
                + " :" + e.getMessage());
    } finally {
        try {
            if (br != null)
                br.close();
        } catch (IOException e) {
            logger.error("Error: " + e);
        }
    }
}

Answer 1

Are you failing to close some resource and relying upon the finaliser thread to pick them up? Or have you just added a finaliser (probably unnecessary) that is preventing a significant amount of memory to be freed promptly.

Answer 2

If that is literally the only change you made, it is difficult to see how that would fix the problem. Whenever the Java Virtual Machine is running low on memory, it runs the garbage collector automatically before throwing an out of memory exception. There should be no need to do this yourself nor any value in doing so.

The only real solution to the problem you describe is to make sure that you clear out any references to objects that are no longer needed. Like if you say:

byte[] ba=new byte[bignumber];
process(ba);

and then you go on and do other things, ba is still sitting there hogging up memory. You want to either exit the function where it was defined, or set ba=null to lose the reference. Then the gc can recycle the memory.

Answer 3

Is there a maximum size to the data you receive (or can you enforce one)? In which case you can declare your byte array outisde the while and reuse it at each iteration without allocating more memory:

...
private static final int BUFFER_SIZE = 102400; //start with a buffer big enough to lower the chances of resizing it -- e.g. 100K
...
msgContent = new byte[BUFFER_SIZE];
while (true) {              
            msgType = dIn.readByte();

            int msgIntType = msgType & 0xff;

                  // get message length

                  int msgIntLen = dIn.readInt();
                  if (msgIntLen != 0) {
                   if( msgIntLen > msgContent.length ) //only resize when needed otherwise reuse
                     msgContent = new byte[msgIntLen];

                   switch(msgIntType) {
            case 4:
            //case MSG_DATA:
                // MSG_DATA
                recordCount++;
                processData(msgContent, msgIntLen); //need to change your function to also pass in the size of the message read!
                if (recordCount == 2000) {
                sendACK(dOut, msgIntType);
                logger.info("sent ACK for MSG_DATA");
                recordCount = 0;
                }               
                break;

}

Answer 4

What is the the type of your dIN variable? Maybe I am misunderstanding, but do you really need to read your input into a byte array, then consider the byte array as a stream, and then read the stream line by line? If you already know the structure of your content, why create all the intermediate steps. You could just as well process(dIn) somehow.

Also, just to confirm, is this running in a multi-threaded environment??