Removing items from a ConcurrentDictionary

Question

I've got Queue of Sky objects where my process picks up each one and executes it. If the results are needed they are put into a ConcurrentDictionary to be retrieved by using a GUID. Sky objects are added and executed from the queue every few milliseconds and this process will run for hours. Retrieving the results from the ConcurrentDictionary does a try remove and removes objects, but some may not be retrieved because they are not needed. I've implemented the idea of not creating the results if they are not needed. As a just in case plan I added a UTC creation date to each sky object.

I want to create a cleanup process that runs every few minutes to find any UTC times older than x minutes and remove them. As I understand the ConcurrentDictionary I should have no problem just iterating through the collection and simply remove them, but I have a few questions before writing my clean up procedure.

Should I run the cleanup process in a separate async task? Will my iterating or removing cause keep any results from being added to the ConcurrentDictionary, like blocking issues?

So, I added a second ConcurrentDictionary. When I add the results to the first dictionary I also add the guid and UTC date to the second. The code below iterates the second and any expired items it finds it removed from the first. I haven't tested this yet so I"m not sure if I can remove from the seconds as I'm iterating the dictionary.

    /// <summary>
    /// Use to clean up an sky results objects that might be left in the results dictionary
    /// This compares the now UTC date to the object created date plus the seconds parameter
    /// </summary>
    /// <param name="seconds"></param>
    public static void CleanSkyResultDictionary(double seconds)
    {
        foreach (var skyresult in SkyCleanupDictionary)
        {
            if (skyresult.Value.AddSeconds(seconds) <= DateTime.UtcNow) continue;
            SkyResultDictionary.TryRemove(skyresult.Key, out _);
            SkyCleanupDictionary.TryRemove(skyresult.Key, out _);
        }
    }

Answer 1

A ConcurrentDictionary can be accessed from multiple threads safely without corruption of the internal dictionary data structures. You therefore only need a single dictionary instance as one thread can be adding to it concurrent with another thread iterating over or removing from it.

Answer 2

1. Data

Having two dictionaries means that now you should really synchronize them, which would partly undo the benefits of using a concurrent dictionary.

I suggest to store the timstamp in the same dictionary. One way to do it is:

class ToStore {
 //Constructor here, or add public sets

 public YourClass Data {get;}
 public DateTime AddedAtUtc {get;} 
 //I would suggest using NodaTime's Instant, but that's out of scope for this question.
}

public void Add(YourClass data )
{
    if (data == null)
    {
       throw new ArgumentNullException(nameof(data ));
    }

    var frame = new ToStore {
        Data = data,
        AddedUtc = DateTime.UtcNow 
    }

    dict.TryAdd(frame.TimestampUtc, frame);
    OnAdd(); // fire and forget
}

If key could be the timestamp, you wound't need the ToStore class which would make it even simpler.

2. Cleanup

I don't know you app, but you could consider cleaning up when adding new elements, rather than on timer.

    public void Add(YourOtherClass data )
    {
        (...)
        OnAdd(); // fire and forget
    }

    private void OnAdd()
    {
        Task.Run(() =>
        {
            CleanUp();
        }).ConfigureAwait(false);
    }

Cleanup is:

        foreach (var kvp in dict.Where(IsStale))
        {
            // Please note that by now the frame may have been already
            // removed by another thread.
            dict.TryRemove(kvp.Key, out var ignored);
        }

where IsStale returns true if the frame is old enough to be removed.

I hope this helps.