如何跟踪长时间运行的并行程序的运行位置

Question

I have a restartable program that runs over a very large space and I have started parallelizing it some. Each Task runs independently and updates a database with its results. It doesn't matter if tasks are repeated (they are fully deterministic based on the input array and will simply generate the same result they did before), but doing so is relatively inefficient. So far I have come up with the following pattern:

    static void Main(string[] args) {
        GeneratorStart = Storage.Load();
        var tasks = new List<Task>();
        foreach (int[] temp in Generator()) {
            var arr = temp;
            var task = new Task(() => {
                //... use arr as needed
            });
            task.Start();
            tasks.Add(task);
            if (tasks.Count > 4) {
                Task.WaitAll(tasks.ToArray());
                Storage.UpdateStart(temp);
                tasks = new List<Task>();
            }
        }
    }

Prior to making the generator restartable, I had a simple Parallel.Foreach loop on it and was a bit faster. I think I am losing some CPU time with the WaitAll operation. How can I get rid of this bottleneck while keeping track of what tasks I don't have to run again when I restart?

Other bits for those concerned (shortened for brevity to question):

class Program {
    static bool Done = false;
    static int[] GeneratorStart = null;
    static IEnumerable<int[]> Generator() {
        var s = new Stack<int>();
        //... omitted code to initialize stack to GeneratorStart for brevity
        yield return s.ToArray();
        while (!Done) {
            Increment(s);
            yield return s.Reverse().ToArray();
        }
    }

    static int Base = 25600; //example number (none of this is important
    static void Increment(Stack<int> stack) { //outside the fact 
        if (stack.Count == 0) {               //that it is generating an array
            stack.Push(1);                    //of a large base
            return;                           //behaving like an integer
        }                                     //with each digit stored in an
        int i = stack.Pop();                  //array position)
        i++;
        if (i < Base) {
            stack.Push(i);
            return;
        }
        Increment(stack);
        stack.Push(0);
    }
}

Answer 1

I've come up with this:

    var tasks = new Queue<Pair<int[],Task>>();
    foreach (var temp in Generator()) {
        var arr = temp;
        tasks.Enqueue(new Pair<int[], Task>(arr, Task.Run(() ={
            //... use arr as needed
        }));
        var tArray = t.Select(v => v.Value).Where(t=>!t.IsCompleted).ToArray();
        if (tArray.Length > 7) {
            Task.WaitAny(tArray);
            var first = tasks.Peek();
            while (first != null && first.B.IsCompleted) {
                Storage.UpdateStart(first.A);
                tasks.Dequeue();
                first = tasks.Count == 0 ? null : tasks.Peek();
            }
        }
    }

...
class Pair<TA,TB> {
    public TA A { get; set; }
    public TB B { get; set; }
    public Pair(TA a, TB b) { A = a; B = b; }
}