Java中的部分反序列化和序列化？

Question

There are a huge number of libraries and approaches out there to serialize and de-serialize objects in Java. What I would like to do involves rather large and complex objects which need to get sent back and forth between processing nodes.

However, each node only is interested in one or a few, usually small parts of the whole object. The processing node processes that part and creates a new part that would need to get spliced into the existing serialized object before it gets sent on.

For this, two things would be of high importance:

• being able to just deserialize parts of the serialized object (and thus save parsing/deserialization time, object creation time, memory...) and to also add the serialization of some new part to the existing serialized object (again saving time and memory) -- skipping the unwanted parts in the serialized version should be extremely fast and efficient and should ideally be possible in a streaming mode, without the need to keep the whole serialized data in memory at once
• overall compact and fast serialization and deserialization.

I am pretty flexible as to how much automation I get for actually creating typed objects versus untyped maps and lists: if all else fails I would be able to represent the whole object as a nested data structure of just maps, arrays and the basic datatypes boolean, String and Number.

UPDATE: forgot to mention two additional, rather important requirements:

• the solution must be possible with the existing objects, ie it is not possible to re-implement the current object using a eg different collections class.

• ideally the solution should be based on open-source software because the software I need this for will be published itself as open-source.

Answer 1

It sounds like you're planning a design where a whole bunch of data is sent to a processing node, and that node will only read/modify/write a small part of it. But then will send the whole bundle on to another node.

Why not have the host that has all the data figure out which node needs which data, and only send that data? Then processing can happen in parallel, instead of daisy-chain. And your total network traffic will be less than every node sending a full copy of everything: O(n*m).

It might be worth designing your own message format, potentially based on JSON, binary, or something else.