Java字符串：内存如何与不可变字符串一起工作

Question

I have a simple question.

byte[] responseData = ...;
String str = new String(responseData);
String withKey = "{\"Abcd\":" + str + "}";

in the above code, are these three lines taking 3X memory. for example if the responseData is 1mb, then line 2 will take an extra 1mb in memory and then line 3 will take extra 1mb + xx. is this true? if no, then how it is going to work. if yes, then what is the optimal way to fix this. will StringBuffer help here?

Answer 1

Yes, that sounds about right. Probably even more because your 1MB byte array needs to be turned into UTF-16, so depending on the encoding, it may be even bigger (2MB if the input was ASCII).

Note that the garbage collector can reclaim memory as soon as the variables that use it go out of scope. You could set them to null as early as possible to help it make this as timely as possible (for example responseData = null; after you constructed your String).

if yes, then what is the optimal way to fix this

"Fix" implies a problem. If you have enough memory there is no problem.

the problem is that I am getting OutOfMemoryException as the byte[] data coming from server is quite big,

If you don't, you have to think about a better alternative to keeping a 1MB string in memory. Maybe you can stream the data off a file? Or work on the byte array directly? What kind of data is this?

Answer 2

The problem is that I am getting OutOfMemoryException as the byte[] data coming from server is quite big, thats why I need to figure it out first that am I doing something wrong ....

Yes. Well basically your fundamental problem is that you are trying to hold the entire string in memory at one time. This is always going to fail for a sufficiently large string ... even if you code it in the most optimal memory efficient fashion possible. (And that would be complicated in itself.)

The ultimate solution (ie the one that "scales") is to do one of the following:

• stream the data to the file system, or

• process it in such a way that you don't need ever need the entire "string" to be represented.

---

You asked if StringBuffer will help. It might help a bit ... provided that you use it correctly. The trick is to make sure that you preallocate the StringBuffer (actually a StringBuilder is better!!) to be big enough to hold all of the characters required. Then copy data into it using a charset decoder (directly or using a Reader pipeline).

But even with optimal coding, you are likely to need a peak of 3 times the size of your input byte[] .

---

Note that your OOME problem is probably nothing to do with GC or storage leaks. It is actually about the fundamental space requirements of the data types you are using ... and the fact that Java does not offer a "string of bytes" data type.

Answer 3

There is no such OutOfMemoryException in my apidocs. If it's OutOfMemoryError , especially on the server-side, you definitely got a problem.

When you receive big requests from clients, those String related statements are not the first problem. Reducing 3X to 1X is not the solution.

I'm sorry I can't help without any further codes.

Use back-end storage

You should not store the whole request body on byte[] . You can store them directly on any back-end storage such as a local file, a remote database, or cloud storage.

I would

copy stream from request to back-end with small chunked buffer

Use streams

If can use Streams not Objects.

I would

response.getWriter().write("{\"Abcd\":");
copy <your back-end stored data as stream>);
response.getWriter().write("}");

Answer 4

Yes, if you use a Stringbuffer for the code you have, you would save 1mb of heap space in the last step. However, considering the size of data you have, I recommend an external memory algorithm where you bring only part of your data to memory, process it and put it back to storage.

Answer 5

As others have mentioned, you should really try not to have such a big Object in your mobile app, and that streaming should be your best solution.

That said, there are some techniques to reduce the amount memory your app is using now:

1. Remove byte[] responseData entirely if possible, so the memory it used can be released ASAP (assuming it is not used anywhere else)

2. Create the largest String first, and then substring() it, Android uses Apache Harmony for its standard Java library implementation. If you check its String class implementation , you'll see that substring() is implemented simply by creating a new String object with the proper start and end offset to the original data and no duplicate copy is created. So doing the following would cuts the overall memory consumption by at least 1/3 :

   String withKey = StringBuilder().append("{\\"Abcd\\").append(str).append("}").toString(); String str = withKey.substring("{\\"Abcd\\".length(), withKey.length()-"}".length());

3. Never ever use something like "{\\"Abcd\\":" + str + "}" for large Strings, under the hood "string_a"+"string_b" is implemented as new StringBuilder().append("string_a").append("string_b").toString(); so implicitly you are creating two (or at least one if the compiler is mart) StringBuilders. For large Strings, it's better that you take over this process yourself as you have deep domain knowledge about your program that the compiler doesn't, and knows how to best manipulate the strings.