Understanding the default Microsoft Orleans Grain Directory

Question

I am attempting to understand the default Orleans Grain Directory.

https://learn.microsoft.com/en-us/do.net/orleans/host/grain-directory

This article strongly suggests that you should use the default directory, and that you should only switch to a different directory if you have specific use cases and reasons for switching.

Question : if the default directory is an eventually consistent in memory store, then there is a guarantee that over time grains will be duplicated. This could be disastrous for application state.

Obviously microsoft has some mitigation for this. I am wondering what it is?

Answer 1

Orleans will ensure that the grain directory converges over time so that there is exactly one registration for each grain activation in the cluster.

Since it is eventually consistent there can be periods where a grain is active and is not registered (eg, because the host which the grain was registered on was just struck by a meteor) and there can be periods where multiple distinct activations for a grain are registered on different hosts (eg, a race where a new host just joined the cluster and not all hosts are aware of that yet).

These cases are both rectified quickly: the directory partitions split and merge so that there are no overlaps (two silos thinking they own some range of keys) or gaps (no silo owning some range of keys) and in doing so, duplicates are resolved by deactivation and activations which are missing a registration are terminated as soon as the gap is discovered (ie, when the silo a grain was registered on has been evicted).

Note that even with a strongly consistent directory it would be possible to have duplicate activations. Consider the case where a silo loses.network connectivity and is subsequently evicted by the other silos in the cluster, or the case where a host performs a very long GC pause, or a VM migration from one host to another. In these cases, it's possible that the silo continues running for some period of time before knowing that it should terminate itself (upon learning that it has been evicted).

Grains should put critical application state in consistent storage. When Orleans saves grain state, it uses optimistic concurrency control to ensure that it does not clobber unobserved writes. Due to this, even if you do have duplicate activations, any write is guaranteed to have seen all previous writes. By default, a write which fails due to inconsistency will also cause a grain to deactivate, which can resolve duplicates even before cluster membership detects that a silo has been evicted.