Consuming 200+ JSON requests with different data structures [on hold]

Question

I'm designing a project where there is a desire to consume multiple inbound JSON requests via Web API, each with differing data structures. These requests would be processed to insert data into a SQL database, typically with a one to many relationship.

I'm figuring out the best way to make the project efficient, without too much overhead in terms of maintenance.

I don't really want 200 end points (it could be 500 later down the line), so thought I would accept payloads through just one POST endpoint. Using a UID in the call, I could use logic to instantiate the correct class to process the inbound request. So in short, this design would be a single end point with 200 classes.

I've also thought about setting up each one as it's own microservice. But then there is the overhead of deploying and maintaining each project.

Essentially this project is acting as a very basic ETL middleware. I was wondering if anyone has any similar experience along these lines already as I'd like to hear how you approached the design and how things went.

Answer 1

This is a very subjective area and people may not agree on the approach. Objectively, and as you outline, there are a number of approaches as you out line. Your idea of one endpoint with multiple classes to process differing JSON payloads makes ok sense. A micro service for each sounds like overkill since you may not want to publish and connect a new endpoint for every new payload.

That said, maybe there is a reasonable middle ground; consider shifting your thinking towards the business domain - out of the 200 services, are some of them related to sales, some to CRM, some to inventory, some to invoicing etc. That way you can find a reasonable grain you can avoid the complexity of "everything in one" and the overhead of 200 endpoints.

Also consider a mapping file or configuration that knows how to convert the JSON file to the database code to store, then you don't need to redeploy each service, just add another configuration as you build out functionality to deal with new JSON files.

Answer 2

You can look at this solution https://github.com/DiyazY/audit . It was written many years ago and maybe needs some updates and refactoring. But the idea is very similar, save any kind of payload in a database(there is a MongoDB). Also, you can save the same payload several times with some changes (you have to provide a guid id), retrieve this object by id and see how changes were and what was changed in time sequence. I'm not sure that it's what you are looking for. But I hope it will help you. And if you find it usable and want to upgrade codebase or make some changes, feel free and push it))

Answer 3

I agree with Murray's point above about considering some kind of grouping based on what the messages "mean" within your domain, so that your API reflects this natural division in some way.

You can kind of eat your cake and have it here, because you can make the "type" of the message part of the endpoint from the client's perspective eg if your route pattern on your POST method was something like

/api/services/{serviceName}/

then your method signature looks something like

public async Task<IActionResult> Post(string serviceName, [FromBody] string payload)
{
    ///use serviceName as an input to your handler factory or whatever
    ///and of course you have to handle the deserialisation of payload yourself.
}

• So you do have one "endpoint" in reality but from the client's perspective at least they are POSTing to "different" endpoints. Of course going with a "_type" member on the payload or using a UUID or whatever is also fine - just don't call it RESTful:)

• As to previous experience with this, yes, I have used an approach like this in the past, in my case it was a web api that was serving as an entry point to an SAP Business One back-end and was basically enabling the SAP SDK (which is a very thin adapter over an SQL connection) to be used over HTTP.

• Like you I had a massive amount of possible types to handle and at the time I figured it was better to build a generic system that let me add handlers for the various types on the fly. So a single endpoint that used metadata in the payload to map to specific handlers. Basically an ETL middleware in that respect - take the payload, figure out the handler (in my case, with a "_type" property on the payload) and pass it off to be transformed (well, this is SAP so "mangled" is probably the right word.) into the right format for offering into the SAP APIs.

• And it worked OK for me, I have to say it was pretty successful and nearly 10 years later is still working for the client.

BUT. If I had to do it again I would consider biting the bullet and creating individual endpoints that handled specific messages.

• I say this because in my case, over time little bits and pieces of logic started to leak into the generic endpoint code (for example: I ask the clients to send a number in a particular field representing an employee ID. But the destination API needs more information than that from the SAP database so I need to use that value as a lookup to get the information. And of course because your mindset is "generalise" you start to find yourself putting this kind of general lookup capability into the endpoint, NOT the type-specific handler. Which is fine until you start to discover that not all the lookups for employeeID need the exact same fields returned and things get messy!)

• So honestly if I had to guesstimate I would say I spent at least as much time mucking around with that as I would have spent separating them out in the first place.

Once you put the little bit of up-front work in, it then becomes much easier in the future to separate our groups of these endpoints into microservices, you are literally cutting off a slice of your API and copying to a new API.

If you consider for example how incredibly easy it is to use something like MediatR with ASP.Net these days, you will realise that even 200 individual handlers is not that much work at all, and combined with something like ReSharper you will spend a couple of hours tops boilerplating out those individual handlers, all you really need to worry about is the mapping of types to handlers.

Hope that helps