Best way to transfer data struct between two ASP.NET Core API's

Question

I have two API's. API-1 generates a huge dataset, something like

Dictionary<string,Dictionary<string<List<object>>> data_

My final goal is to find the most efficient way to transfer this data to API-2 deserialize it and return response to API-1.

I have found two approaches:

1. To generate JSON form the container and send the JSON to API-2. Here we have problem if the data is too big.
2. To save the JSON to a file and send the file API-2. Here we have a problem because the file usually gets like 400-500 mb in size.

So my question is, is there a way to transfer directly the data struct to API-2, without JSON serialization/deserialization.

Is there some way to send my data_ container like bytes, and serialize it in API-2?

Answer 1

From what you described, it sounds that this approach of sending all the data in one API call is not suited here.

I suggest that you will send the data to API-2 in chunks.

In API-1 you will save the records with indication of [sent/not sent] in a database. Then you should loop through the records, get chunks of them, and send the chunks to API-2. When you get success from API-2, you should mark the records as "sent" in API1 DB.