如何使用StringBuilder读取Java中的大文本文件？

Question

Is there any mechanism in Java to reduce the memory usage while reading large text files?

Almost every program I've come across uses String to read text files.But Java reserves space for each String literal.That's why I think memory usage gets increased since all String objects are stored. All the classes of java.io deals with String. But if we're not using StringBuilder then how can we reduce memory usage?

After all reducing memory usage is the primary concern of StringBuilder[since it's not immutable like String]. Then how can we exploit its feature in Java I/O operation without using String ie without using something like this: sb.append([String object]);

Answer 1

Assume you have n strings, each of length 1 that you read from your input - for simplicity.

Using operator+ on strigns while reading will create a String object each time you concatenate strings, so you get strings of length 1,2,3,...,n

So the total memory usage of the strings combined is 1 + 2 + .. + n = O(n^2) in addition to the n strings you read from input

while if you use StringBuilder to create the final string, you actually create n - for input [each of length 1] and one object for the final string - of size n , so total memory usage of 1 + 1 + .. + 1 + n = O(n)

So, even if you use sb.append(String) - the space usage is asymptotcally beter then creating all intermediate strings - since you do not need to create intermediate String objects.

In addition - the performance [time] should be better when using StringBuilder - both because you create less objects, and both because of lesser memory usage - the gc doesn't need to work as hard as when concatenating strings naively.

(*)Note that it is easy to see that the above still holds for any length of strings.

Answer 2

您可以使用StringBuilders的append char方法来避免创建中间字符串，请参阅以下文章： https : //stackoverflow.com/a/9849624/102483请记住，没有任何方法可以减少内存的占用空间。 final String，以使其小于您正在读取的文件的大小。

Answer 3

Depending on what you are doing, you could create a pool of String and/or StringBuilder objects that are loaded with the values you need, cleared out and then reused. You could configure the pool to grow to a maximum value, and if the objects in the pool are not used, then set them to null where they will eventually be reclaimed by the garbage collector.

Answer 4

You might want to consider something like this:

  BufferedReader reader = 
    new BufferedReader(
      new InputStreamReader(
        new ByteArrayInputStream(data)));
  String line;

  while ((line = reader.readLine()) != null)
    ...

See these links for more details:

BufferedReader for large ByteBuffer?

http://www.tutorialspoint.com/java/java_bytearrayinputstream.htm

Answer 5

Reader and its subclasses are based around char and char[], only convenience methods use String. Since StringBuilder.append() accepts char[] you can avoid creating unnecessary String objects if you only use the methods build around char[].

Note that while this reduces the number of temporary created String objects the overall memory requirements stay the same, the gc would collect any otherwise created String.

Answer 6

Instead of String, try using StringBuilder to append data read from a file. If you use String you might end up creating multiple string objects in memory.