使用Java复制流（模拟telnet）时发生严重的内存泄漏

Question

below link is a Apache commonsnet example of Telnet, it simulate the user using telnet client(type the telnet command and get output), one thread is reading stream and one thread is writing stream.

http://www.docjar.com/html/api/examples/weatherTelnet.java.html

i modify this program, the user can login in the telnet server and type command but never logged out. about every 15 second, the user type a command and the server give him output. i use another thread to copy the server output to local file, but the program can work, but it will cause serious memory leak, about a few hours later the program quit because of OutofMemoryException, using Java heap memory dump tool i can see the root cause is char[] which is definitely caused during copy stream.

Could somebody help me point out where the problem is and how to fix? thanks very much.

 public static final void readWrite( final InputStream remoteInput, final OutputStream remoteOutput,
                                     final String address ) throws FileNotFoundException, InterruptedException
{
    Thread reader = null;
    Thread writer = null;


    final String[] commands = new String[]{ "username\n", "password\n", "command\n" };


        reader = new Thread("reader")
        {
            @SuppressWarnings( "null" )
            @Override
            public void run()
            {
                String currentCommand = null;
                try
                {
                    int i = 0;
                    while( !interrupted() )
                    {
                        if( i >= 3 )
                            currentCommand = "command\n";
                        else
                            currentCommand = commands[i++];
                        remoteOutput.write( currentCommand.getBytes() );
                        remoteOutput.flush();
                        sleep( 15000 );
                    }
                }
                catch( IOException e )
                {
                    e.printStackTrace();

                    this.interrupt();
                }
                catch( InterruptedException e )
                {

                    this.interrupt();
                }
            }
        };


        writer = new Thread("writer" )
        {
            @Override
            public void run()
            {
                try
                {
                    while( !isInterrupted() )
                    {
                        FileOutputStream fos = new FileOutputStream( "logfile" );
                        copyStream( remoteInput, fos, 1024, false );
                        fos.close();
                        sleep( 10000 );
                    }
                }
                catch( IOException e )
                {

                    this.interrupt();
                }
                catch( InterruptedException e )
                {

                    this.interrupt();
                }
            }
        };
        writer.setPriority( Thread.currentThread().getPriority() + 1 );
    }
    reader.start();

    Thread.sleep( 2000 );

    writer.start();
   Thread.currentThread().join();
}

public static final long copyStream( InputStream source, OutputStream dest, int bufferSize, boolean flush )
    throws CopyStreamException
{
    int bytes;
    long total;
    byte[] buffer;

    buffer = new byte[ bufferSize ];
    total = 0;

    try
    {
        while( ( bytes = source.read( buffer ) ) != -1 )
        {
            // Technically, some read(byte[]) methods may return 0 and we cannot
            // accept that as an indication of EOF.

            if( bytes == 13 )
            {
                dest.flush();
                break;
            }
            if( bytes == 0 )
            {
                bytes = source.read();
                if( bytes < 0 )
                    break;
                dest.write( bytes );
                if( flush )
                    dest.flush();
                ++total;
                continue;
            }

            dest.write( buffer, 0, bytes );
            if( flush )
                dest.flush();
            total += bytes;
        }
    }
    catch( IOException e )
    {
        throw new CopyStreamException( "IOException caught while copying.", total, e );
    }

    return total;
}

Responding to Peter Lawrey's Answer

The comment area is too limited please let me type it here.

• The whole application will exist until the last non-daemon thread exits.

  yes, I should set one of the thread daemon without using the main thread join(), right?

• You don't need to set priorities, they usually don't do what you think.

  OK

• There is no point interrupting a thread which is about to return/finish.

  Yes, should remove this Interrupt(); it's nonsense.

• Why would you flush the data whenever you read exactly 13 bytes?

  Well it's a bad design beause of my real demand, but that would be another topic, see later about this.

• Your 0 length data won't happen for a blocking connection, but if it did, your handling isn't correct and you better off without it.

  Well, actually this method is copied from Apache Commons net, see here: org. Apache. Commons. Net. Io. Util. Copystream(... ) org. Apache. Commons. Net. Io. Util. Copystream(... )

• CopyStream copies the data until the stream ends, however you have this in a loop which means 10 seconds later you will truncate the file and replace it with no data (as the stream is still closed and you don't append the log file)

  Yes that's what I need, I only need the latest command output, previous one has no meaning to me, if I didn't truncate the file, soon the file will become bigger and bigger.

• Your "reader" writes data, it doesn't read anything. Your "writer" thread both reads and writes.

  Yes maybe not a good name, I'll change it later.

• You are using Thread directly which is considered bad practice. You should use a Runnable and wrap it with a Thread or use an ExecutorService

  Yes, I'll change it later.

• You catch a number of exceptions but don't print them out. It is brave to assume you don't need to know when they are thrown as your thread might die silently and you won't know why.

  Yes, I'll modify it later and have logs in it.

• I assume CopyStreamException is an Exception, in which case you have gone to the trouble of creating a nice exception to wrap the IOException, which you later throw away. You don't need to sleep between starting the reader and the writer.

  Yes I think so.

• As your current thread is just waiting for two other threads you could use the current thread to do the writing "reader" and have just one background logging thread.

  Yes only for standalone application, in fact this piece of code is extracted from a web application(server side), the main thread has another job, I hope the two thread I made never die(keep running when the web server running)

• The method doesn't throw FileNotFoundException.

  It throws, see the outer method declaration.

• You calculate the total data copied, but it is never used.

  Yes should be removed.

• To copy a stream I suggest you use IOUtils. Copy(InputStream, OutputStream)

  I used before, however it did not meet my need: it will block all the time, and the log file's size soon get too huge that I have to use a customized one.

• Why read 13 byte and quit?

  Surely it's a magic number. I found if 13 bytes is read then it reach to the output's end. Obvious it's a bad design, could you help me found a more NICE way?

• When the former thread type a command, the server will give back about 300Kb content.

  I need a method which can detect the end of server output, but I fail to figure it out because the while( ( bytes = source. Read( buffer ) )!= -1 ) just blocked!

Answer 1

Your program doesn't have a char[] so if you are getting an OutOfMemoryError creating a char[] can to show us the line of code which does this?

I was going to write a comment, but it was far too long.

• The whole application will exist until the last non-daemon thread exits.
• You don't need to set priorities, they usually don't do what you think.
• There is no point interrupting a thread which is about to return/finish.
• You flush when you read exactly 13 bytes, but you never send exact 13 bytes.
• Your 0 length data won't happen for a blocking connection, but if it did, your handling isn't correct and you better off without it.
• copyStream copies the data until the stream ends, however you have this in a loop which means 10 seconds later you will truncate the file and replace it with no data (as the stream is still closed and you don't append the log file)
• your "reader" writes data, it doesn't read anything.
• your "writer" thread both reads and writes.
• you are using Thread directly which is considered bad practice. You should use a Runnable and wrap it with a Thread or use an ExecutorService
• you catch a number of exceptions but don't print them out. It is brave to assume you don't need to know when they are thrown as your thread might die silently and you won't know why.
• I assume CopyStreamException is an Exception, in which case you have gone to the trouble of creating a nice exception to wrap the IOException, which you later throw away.
• you don't need to sleep between starting the reader and the writer.
• as your current thread is just waiting for two other threads you could use the current thread to do the writing "reader" and have just one background logging thread.
• the method doesn't throw FileNotFoundException.
• you calculate the total data copied, but it is never used.
• To copy a stream I suggest you use IOUtils.copy(InputStream, OutputStream)

Answer 2

Are you that the leak is from the copyStream method? How did you determine that? To echo @Peter Lawrey initial statement, the only places in your code that refer to char[] are the actual Strings created. Since your program runs for a long time, always creating Strings to be processed, and assuming these are unique Strings (ie, not reused by the JVM's String pool, but a new String created for each), that could explain your memory usage. You may simply have a lot of Strings...