Automatic Retry on grain fail-over in Microsoft Orleans

Question

so we're testing out how Grain Fail-over works in case some silo is unresponsive for some reason (server is down, etc...). Currently we have two silos running on two different machines with grains activated on each of them. We then proceed to kill one of the silos (unexpectedly). We try to call a grain on the DEAD silo and after some timeout (I believe it is 3 minutes or so in total) an Exception is thrown, stating that the Silo is rejecting the connection. Now, we believe that after a silo is declared DEAD a grain is only activated on another silo if we retry the activation. This is working fine for us. However, we would like to know if there is some way to do the retry automatically instead of doing the logic ourselves.

Answer 1

First, the 3 minutes sounds way too much. It should be tens of a second, if you are using the default liveness settings. What system store is it?

If you want to automatically retry, you better wrap all your client grain calls in a wrapper which will retry with exponential back off for example. Doing retries yourself gives you much more control, and what to retry and how.