Service fabric actors state

Question

We are planning to use Service fabric actor model for one of our user services. We have thousands of users and they have their own profile data. By far reading the materials, service fabric actor model maintains its states with their service fabric cluster. I couldn't get a clear picture in disaster recovery/planned shutdown scenarios/offline data access. In such cases, Is it needed to persist the data out side of these actor service?

What happens to the data, if we decided to shutdown all the service fabric cluster one day, and wanted to reactivate few days later?

Answer 1

In an SF cluster in Azure, the data is stored on the temp drive. There's no guarantee that a node that is shutdown retains the temp drive. So shutting down all nodes simultaneously will result in data loss.

To avoid this, you should regularly create backups of your (Actor) Services. For instance by using this Nuget package. Store the resulting files outside the cluster.

The cluster technology will help keep your data safe during failures of nodes, eg in a 5 node cluster, 4 remaining healthy nodes can take over the work of a failed node. Data is stored redundantly, so your services remain operational. The same functionality also allows for rolling upgrades of services/actors.

Here's an article about DR.

Answer 2

I had implemented a large enterprise application in service fabric using actor model for management of orders. Few things that might help while choosing a strategy for data backup and restoration

1. As the package https://github.com/loekd/ServiceFabric.BackupRestore is not full fledged and you need to take care of some of the scenario. for example: During deployment your actor partitions moved to other nodes and if you try to take incremental backups it will failed with FabricMissingFullBackupException because on that node after becoming primary you haven't took the Full backup and some one needs to manually fix the issue. How we added the retry pattern to fix that issue is not in the scope of this question.
2. Incremental backups didn't restore always during restoration process.
3. Some time Incremental backup creation failed even if you set the logTrunctationIntervalInMinutes properly.
4. Some developer by mistake deleted the service or application you will loss all your data.
5. if your system heavily dependent on Reminder's which was in our case. During restoration all the reminders gets reset.

Good Solution: Override the default KvsActorStateProvider with your own implementation which stores the data in DocumentDB, MongoDB, Cassandra or Azure SQL if you want to use the power BI for some analytics.