从 Java 服务到 Java 服务通过 RestTemplate 将文件作为字节 [] 发送时，是否需要额外的 base64 编码？

Question

I am sending data via json body in a post request from a client (Java) to a server (Java) using a Spring RestTemplate and RestController.

The data is present as a POJO on the client and will be parsed into a POJO with the same structure on the server.

On the client I am converting a file with Files.readAllBytes to byte[] and store it in the content field.

On the server side the whole object including the byte[] will be marshalled to XML using JAXB annotations.

class BinaryObject {
  String fileName;
  String mimeCode;
  byte[] content;
}

Everything is working fine and running as intended. I heard it could be beneficial to encode the content field before transmitting the date to the server and decode it there before it is marshaled into XML.

My Question

Is it necessary or recommended to additionally encode / decode the content field with base64?

Answer 1

TL;DR

To the best of my knowledge, you are not going against any good practice with your current implementation. One might question the design (exchanging files in JSON? Storing binary inside XML?), but this is a separate question.

Still, there is room for possible optmization, but the toolset you use (eg Spring rest template + Spring Controler + JSON serialization (jackson) + XML using JAXB) kind of hide the possible optimizations from you.

You have to carrefully weight the pros and cons of working around your comfortable "automat(g)ical" serializations that work well as of today to see if it is worth the trouble to tweak it.

We can nonetheless discuss the theory of what could be done.

A discussion about Base64

Base64 encoding in an efficient way to encode binary data in pure text formats (eg MIME strucutres such as email or some HTTP bodies, JSON, XML, ...) but it has two costs: the first is a non negligible size increase (~ 33% size), the second is CPU time.

Sometimes, (but you'd have to profile, check if that is your case), this cost is not negligible, esp. for large files (due to some buffering and char/byte conversions in the frameworks, you could easilly end up using eg 4x the size of the encoded file in the Java Heap).

When handling 10kb files at 10 requests/sec, this is usually NOT an issue. But 10MB files at 100 req/second, well that is another ball park.

So you'd have to check (I doubt your typical server will reach 100 req/s with 10MB files, because that is a 1GB/s incoming network bandwidth).

What is optimizable in your current process

In your current process, you have multiple encodings taking place: the client needs to Base64 encode the bytes read from the file.

When the request hits the server, the server decodes the base64 to a byte[] , then your XML serialization (JAXB) reconverts the byte[] to base64.

So in effect, "you" (more exactly, the REST controler side of things) decoded base64 content, all for nothing because the XML side of things could have used it directly.

What could be done

A few things.

Do you need base64 at the calling site?

First, you do not have to encode at the client side. When using JSON, there is no choice, but the world did not wait for JSON to exchange files (eg arbitrary binary content) over HTTP.

If your content is a file name, a MIME type, and a file body, then standard, direct HTTP calls with no JSON at all is perfectly fine.

The MIME type could be mapped to the Content-Type HTTP Header, the file name inside the Content-Disposition HTTP header, and the contents as the raw HTTP body. No base64 needed (but you need your server-side to accept raw HTTP content as is). This is standard as can be.

This change would allow you to remove the encoding (client side), lower the network size of the call (~33% less), and remove one decoding at the server side. The server would just have to base64 encode (once) a raw stream to produce the XML, and you would not even need to buffer the whole file contents for that (you'd have to tweak you JAXB model a bit, but you can JAXB serialize directly bytes from an InputStream , which means, almost no buffer, and since your CPU probably encodes faster than your network serves content, no real latency incurred).

If this, for some reason, is not an option, let's say your client has to send JSON (and therefore base64 content)

Can you avoid decoding at the server side

Sort of. You can use a server-side bean where the content is actually a String and NOT a byte[] . This is hacky, but your REST controler will no longer deserialize base64, it will keep it "as is", which is a JSON string (which happens to be base64 encoded content, but the controler does not care).

So your server will have saved the CPU cost of one base64 decoding, but in exchange, you'll have a base64 String in java heap (compared to the raw byte[] , +33% size on Java >=9 with compact strings, +166% size on Java < 9).

If you are to profit from this, you also have to tweak your JAXB to see the base64 encoded String as a byte[] , which is not trivial as far as I can tell, unless you modify the JAXB object in such a way that it accepts a String instead of the byte[] which is kind of hacky (if your JAXB objects are generated from a XML schema, this might really become a pain to implement)

All in all this is much harder - probably too much if you are not really hitting the wall, performance wise, on this particular issue.

A few other stuff

Are your files pure binary, or are they actually text? If there are text, you may benefit from using CDATA encoding on the XML side instead of base64?

Is your XML actually a SOAP call? If so, and if the service supports MTOM, you could avoid base64 completely, but that is an altogether different subject.