Azure Service Fabric Scaling

Question

I have an app that runs scheduled tasks on Azure Service Fabric. My app must run thirty to forty tasks at the same time, so I am using asynchronous programming. I have some questions:

Do you recommend running the tasks asynchronously? If not, should I run the task synchronously and scale up? How do I scale up? I need no return information from running the task.

Should I separate the queuing and dequeuing into separate stateful services? If so, how would these services communicate to each other?

Here is my code:

internal sealed class JMATaskRunner : StatefulService
   {
    public JMATaskRunner(StatefulServiceContext context)
        : base(context)
    { }

    /// <summary>
    /// Optional override to create listeners (e.g., HTTP, Service Remoting, WCF, etc.) for this service replica to handle client or user requests.
    /// </summary>
    /// <remarks>
    /// For more information on service communication, see http://aka.ms/servicefabricservicecommunication
    /// </remarks>
    /// <returns>A collection of listeners.</returns>
    protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
    {
        return new ServiceReplicaListener[0];
    }


    public async Task<List<JMATask>> GetMessagesAsync()
    {
        await AddTasks();

        List<JMATask> ts = new List<JMATask>();

        IReliableQueue<JMATask> tasks =
            await this.StateManager.GetOrAddAsync<IReliableQueue<JMATask>>("JMATasks");

        using (ITransaction tx = this.StateManager.CreateTransaction())
        {
            var messagesEnumerable = await tasks.CreateEnumerableAsync(tx);

            using (var enumerator = messagesEnumerable.GetAsyncEnumerator())
            {
                while (await enumerator.MoveNextAsync(CancellationToken.None))
                {
                    ts.Add(enumerator.Current);
                }
            }
        }

        return ts;
        //return messagesEnumerable.ToList();
    }

    public async Task AddMessageAsync(JMATask task)
    {
        IReliableQueue<JMATask> tasks =
            await this.StateManager.GetOrAddAsync<IReliableQueue<JMATask>>("JMATasks");

        using (ITransaction tx = this.StateManager.CreateTransaction())
        {
            await tasks.EnqueueAsync(tx, task);
            await tx.CommitAsync();
        }
    }

    /// <summary>
    /// This is the main entry point for your service replica.
    /// This method executes when this replica of your service becomes primary and has write status.
    /// </summary>
    /// <param name="cancellationToken">Canceled when Service Fabric needs to shut down this service replica.</param>
    protected override async Task RunAsync(CancellationToken cancellationToken)
    {
        // TODO: Replace the following sample code with your own logic 
        //       or remove this RunAsync override if it's not needed in your service.

        IReliableQueue<JMATask> tasks =
            await this.StateManager.GetOrAddAsync<IReliableQueue<JMATask>>("JMATasks");

        while (true)
        {
            cancellationToken.ThrowIfCancellationRequested();

            var messagesEnumerable = await GetMessagesAsync();

            using (ITransaction tx = this.StateManager.CreateTransaction())
            {
                foreach (var message in messagesEnumerable)
                {
                    var result = await tasks.TryDequeueAsync(tx);
                    await PerformTask(result.Value);
                }

                await tx.CommitAsync();
                await Task.Delay(TimeSpan.FromSeconds(2), cancellationToken);
            }
        }


    }

    async Task<JMATask> PerformTask(JMATask task)
    {
        await Task.Run(() => Perform(task));
        return task;
    }

    void Perform(JMATask task)
    {
        Thread.Sleep(50000);
    }

    async Task<JMATask> AddTasks()
    {
        m_TaskProvider = JMATaskFactory.Get(conString);

        //List<JMATask> tasks = m_TaskProvider.GetAllTasks();

        //foreach(JMATask task in tasks)
        //{
        //    await AddMessageAsync(task);
        //}

        JMATask task = m_TaskProvider.GetJMATask(80);
        JMATask task2 = m_TaskProvider.GetJMATask(97);

        await AddMessageAsync(task);
        await AddMessageAsync(task2);
        return new JMATask();
    }
}

Answer 1

For service fabric (and other actor-based systems), you typically want to scale out as opposed to scaling up [Scaling Up vs Scaling Out] see ( http://www.vtagion.com/scalability-scale-up-scale-out-care/ )

Azure Article on scaling up and down (slight misnomer).

Essentially, service fabric takes care of most of the concerns around failover, load balancing etc.

The documentation on how to scale the clusters is well worth a read.

The documentation on reliable actors also goes into the "threading" model of the system, which as it's actor-based is fundamentally asynchronous.

Answer 2

You can certainly do asynchronous tasks, but when you create more tasks than physical threads available on a machine you'll start seeing diminishing returns.

Service Fabric allows you to scale workloads like this out across multiple machines very easily - think of your machines as a pool of resources. If each VM has 4 physical threads, then with 5 machines you can have a pool of 20 physical threads.

The way to do this is with a partitioned stateful service. Each partition of the service handles a subset of the total workload, which is distributed based on a partition key for each unit of work that you create. See here to get started with partitioning: https://azure.microsoft.com/en-us/documentation/articles/service-fabric-concepts-partitioning/

Answer 3

I would try to use actor model and move away from the reliable queue. Whenever a task needs to be added, it will ask the service fabric runtime give an new instance of actors the represent the logic of task. the service fabric will handle the distribution of the actors and its lifetime. if you need the return value in the future, you can use the stateful service or another aggregate actor to get it. In addition, you will have a little more control over the task.