高性能异步监控任务

Question

I have a couple of hundred devices and I need to check their status every 5 seconds.

The API I'm using contains a blocking function that calls a dll and returns a status of a single device

string status = ReadStatus(int deviceID); // waits here until the status is returned

The above function usually returns the status in a couple of ms, but there will be situations where I might not get the status back for a second or more! Or even worse, one device might not respond at all.

I therefore need to introduce a form of asynchronicity to make sure that one device that doesn't respond doesn't impend all the others being monitored.

My current approach is as following

// triggers every 5 sec
public MonitorDevices_ElapsedInterval(object sender, ElapsedEventArgs elapsedEventArgs)
{       
   foreach (var device in lstDevices) // several hundred devices in the list
   {
       var task = device.ReadStatusAsync(device.ID, cts.Token);
       tasks.Add(task);
    }

   // await all tasks finished, or timeout after 4900ms
   await Task.WhenAny(Task.WhenAll(tasks), Task.Delay(4900, cts.Token));
   cts.Cancel();

   var devicesThatResponded = tasks.Where(t => t.Status == TaskStatus.RanToCompletion)
        .Select(t => t.GetAwaiter().GetResult())
        .ToList();
}

And below in the Device class

public async Task ReadStatusAsync(int deviceID, CancellationToken tk)
{  
    await Task.Delay(50, tk);   
    // calls the dll to return the status. Blocks until the status is return     
    Status = ReadStatus(deviceID);
}

I'm having several problems with my code

1. the foreach loops fires a couple of hundred tasks simultaneously , with the callback from the Task.Delay being served by a thread from the thread pool, each task taking a couple of ms.

I see this as a big potential bottleneck. Are there any better approaches?

This might be similar to what Stephen Cleary commented here, but he didn't provide an alternative What it costs to use Task.Delay()?

2. In case ReadStatus fails to return, I'm trying to use a cancellation token to cancel the thread that sits there waiting for the response... This doesn't seem to work.

   await Task.Delay(50, tk)
   Thread.Sleep(100000) // simulate the device not responding

I still have about 20 Worker Threads alive (even though I was expecting cts.Cancel() to kill them.

Answer 1

the foreach loops fires a couple of hundred tasks simultaneously

Since ReadStatus is synchronous (I'm assuming you can't change this), and since each one needs to be independent because they can block the calling thread, then you have to have hundreds of tasks. That's already the most efficient way.

Are there any better approaches?

If each device should be read every 5 seconds, then each device having its own timer would probably be better. After a few cycles, they should "even out".

await Task.Delay(50, tk);

I do not recommend using Task.Delay to "trampoline" non-async code. If you wish to run code on the thread pool, just wrap it in a Task.Run :

foreach (var device in lstDevices) // several hundred devices in the list
{
  var task = Task.Run(() => device.ReadStatus(device.ID, cts.Token));
  tasks.Add(task);
}

I'm trying to use a cancellation token to cancel the thread that sit there waiting for the response... This doesn't seem to work.

Cancellation tokens do not kill threads. If ReadStatus observes its cancellation token, then it should cancel; if not, then there isn't much you can do about it.

Thread pool threads should not be terminated; this reduces thread churn when the timer next fires.

Answer 2

As you can see in this Microsoft example page of a cancellation token , the doWork method is checking for cancellation on each loop. So, the loop has to start again to cancel out. In your case, when you simulate a long task, it never checks for cancellation at all when it's running.

From How do I cancel non-cancelable async operations? , it's saying at the end : "So, can you cancel non-cancelable operations? No. Can you cancel waits on non-cancelable operations? Sure… just be very careful when you do.". So it answers that we can't cancel it out.

What I would suggest is to use threads with a ThreadPool, you take the starting time of each one and you have an higher priority thread that looks if others bypass their maximum allowed time. If so, Thread.Interrupt() .