Throttling grains in Microsoft Orleans

Question

I am trying to decide on the correct architecture with Microsoft Orleans. I need to retrieve between 1 million and 3 million files from about 1000 systems and store them on a few central servers. The system will also retrieve and store some metadata per file and store them in a database.

I'm currently thinking of using a grain for each file so I'd have potentially millions of grains but I'm wondering if having each grain save to the database concurrently would overload the database in this scenario.

I'm wondering if I should consider one of the following scenarios instead to minimize the concurrent load on the database:

1. Have the grains return their results to another grain which will manage the database interaction and store the metadata in batches.
2. Write the client code in such a way as to throttle the creation of the grains... maybe create/activate grains in batches of 1000 and only create or activate more when the previously created or activated ones have been deactivated and unloaded by the runtime.

Do I need to bother with these or can I simply rely on the Orleans runtime (maybe via a setting) not to activate too many grains at once that would be trying to save data to the database at the same time?

Answer 1

I would throw out the second suggestion because you would be throttling the throughput of the entire system due to difficulty in writing out the results. This is fighting one of the biggest advantages of Orleans, which is virtually unlimited scalability in response to demand.

You could try a modification of the first solution. I wouldn't return results to another grain, but rather have each grain call a stateless service with the results of its file read, then delegate persistence to that service. This may have been what you had in mind, but a stateless service is different from a grain, so I wanted to be clear.

This opens the door to loss of data if the system goes down while this service has a list of rows it is trying to write. If that's a problem, I would have the grains write to a service which immediately writes to a durable message queue(ie RabbitMQ), then another service reads from that queue and writes to the DB. I would not have the grains write directly to the message queue because centralizing the queue writes lets you add retry logic, circuit breakers, etc. to that service. Trying to add a shared circuit breaker to 1MM grains would be a nightmare.

Though you didn't ask, I'm going to throw in another idea. I can't help but question the decision to put all this data into a DB. A DB is very good at keeping track of changes to data, but my suspicion is that the meta data you are collecting will be analyzed later but never modified. That makes this a good candidate for an event stream such as Azure Event Grid. Event streams are optimized for writes and can easily handle the type of surge you are talking about.