Transmitting large data between two spring boot app

Question

I have two spring boot services names A and B.

• Service A receives a multipart file (which is huge in size > 100MB - 1GB in size).
• Service A needs to transmit the file to Service B ( problem area )
• Service B performs some operation on data and return to Service A.
• Calls from A to B is sent using RestTemplate.

Problem Area

• Service A doesn't load the whole file in memory, but read from Multipart InputStream in chunks.
• Chunks are sent to B, rather a complete stream.

Is it possible to send the complete InputStream of multipart to service B, so that service A doesn't load the file as well as chunk the file over http/...(any other way)

(One way I had solved it to use fifo on the underlying linux platform and service A is writing continuously on a pipe and Service B is reading) - was looking someway to achieve it with http if possible.

Background - I want Service B to be in control of how much chunk they want and not Service A. It is because of the operation Service B perform - service B adds some additional bytes, so that when the reverse operation is performed on next data, it can see these additional bytes to understand how much more chunk should be read.

UPDATE OF THE GITHUB I am not sure where I am getting it wrong? I pasted the complete project - https://github.com/robin-carry/large-data-transfer

The main files on client/server where all logic is below / README.md explains everything I am trying/failing to do...

client/src/main/java/com/lockdown/lazy/client/controller/UploadController.java server/src/main/java/com/lockdown/lazy/server/controller/OpServiceController.java

Answer 1

Apache Http Client documentation talks about Request/Response entity streaming - see if that helps solve your scenario.

Answer 2

You can design interface with org.springframework.web.multipart.MultipartFile as argument, then- use it as feign client on sender side and implement it in controller on receiver side