Writing huge string data to file in java, optimization options

Question

I have a chat like desktop java swing app, where i keep getting String type data. Eventually the String variable keeps growing larger and larger.

1) Is it wise idea to keep the large variable in memory and only when the logging is finished save this to disk.

2) If not, then should i continue saving everytime i get a new string (of length about 30-40).

How should i go about optimizing such a desgin?

Answer 1

I would use a BufferedWriter, like PrintWriter. This will buffer the data for you and write every 8 KB (actually every 8192 characters). If you want to write more often you can use flush() or a smaller buffer.

PrintWriter pw = new PrintWriter("my.log");

// will actually write to the OS, 5 times. (1000 * 40 / 8192)
for(int i = 0; i < 1000; i++) {
   pw.printf("%39d%n", i); // a 40 character number.
}

pw.flush();

or you can use

pw.println(lineOfText);

---

BTW: If you want to know what a really huge file looks like ;) This example writes an 8 TB file http://vanillajava.blogspot.com/2011/12/using-memory-mapped-file-for-huge.html

Answer 2

Perhaps you should use a StringBuilder . Append each new message to it, and at the end convert it to a string.

For example,

StringBuilder sb = new StringBuilder();  
// Do your code that continuously adds new messages/strings.
sb.append(new_string);  
// Then once you are done...
String result = sb.toString(); 

If you were to have some string, say String message , and every time you got a new message/string you did message += new_string , it will eat up more memory.

As suggested by Viruzzo, only save so much, then discard the earlier strings at some point. Don't hold on to every message forever.