Store and forward design pattern with multiple instances of a microservice

Question

I am designing a service that will be receiving requests which must only be executed exactly once (can be received more than once due to unique id presence).

The flow is:

outside world -> HTTP -> [A set of instance of Myservice] -> HTTP -> Vendor services

So, I am receiving a bunch of requests over HTTP and I am looking to do some transformation, checking, store the request in a database and send to to the vendor services via HTTP too.

My question is:

What is an effective way to achieve a store and forward pattern in my service ie receive the request and store it immediately, therefore acking it to the "outside world" immediately and then in my own time forward it to the destination service, retrying as many times as needed?

---

Problems:

• What if one of the instances of myservice goes down while it is sending the request to the vendor service?
• How will I monitor if there are a bunch of request that have not been forwarded?
• How to avoid another orchestration service?
• How to avoid another single instance service that is effectively watching the database and forwarding any requests that have not been forwarded.
• How to minimise external dependencies - eg I am aware some sort of queue here might help this issue, but I am trying to see if I can avoid that.

Answer 1

I would advise against NIH syndrom and take a dependency on a mature technology that delivers all your reliability requirements out of the box.

Cadence Workflow is capable of supporting your use case with minimal effort.

Cadence offers a lot of features that are hard to match building custom task orchestration solution:

• Built it exponential retries with unlimited expiration interval
• Failure handling. For example it allows to execute a task that notifies another service if both updates couldn't succeed during a configured interval.
• Support for long running heartbeating operations
• Ability to implement complex task dependencies. For example to implement chaining of calls or compensation logic in case of unrecoverble failures ( SAGA )
• Gives complete visibility into current state of the update. For example when using queues all you know if there are some messages in a queue and you need additional DB to track the overall progress. With Cadence every event is recorded.
• Ability to cancel an update in flight.
• Distributed CRON support

See the presentation that goes over Cadence programming model.