文件访问和 Parallel.For

Question

I have a set of nested methods that effectively copy a file for backup and then make changes to that file using the "RecordConsolidator" class. In the chain of events I get random exceptions that the file is in use which doesn't make sense to me unless every single line of code in a Parallel.For call is executed asynchronously. Here is an example of my current issues (in commented lines):

// this is from a method called CompactFormIDs
try
{
    // will this ever be executed twice on the same object?  Will it be 
    // released before the next line of code after the catch statement?
    File.Copy(dest, source); 
}
catch (Exception e)
{
    weirdExceptions.Add(source); // I keep getting a message that the file already exists
    // even though there is only one copy statement which copies the file above.
}

// creates an undo step in a batch file
Globals.AddCommit(CommitType.RestoreBackup | CommitType.UndoDeleteBackup | CommitType.CommitDeleteBackup, source, dest); 

HashSet<FormID> npcList = new HashSet<FormID>();
uint mask = (uint)masters.Count << 24;
report.BeginAppendProcess();

// the method below also causes an exception.
// It acts as though the file copied is still in use.  No other process accesses the file other than the 
// copy process before this statement.  When not doing Parallel.For this works just fine.
using (RecordConsolidator consolidator = new RecordConsolidator(source, dest, mask, npcList))
{...
}

The end goal is to:

1. Make a copy of the file that will be modified so that it can be restored if things don't work properly with the new version of the file.

2. Add the restoration of the original file to a batch script

3. Make the changes to the file.

How can I do this in a parallel process using a Parallel.For method without encountering all these "file in use/exists" issues. The fact that there is even an issue here makes no sense because the single Copy statement is causing multiple issues that shouldn't happen unless the copy isn't complete before the rest of the code is executed or somehow Parallel.For is being executed twice for each item.

UPDATE 1: This is the method that contains the Parallel.For loop:

private void OnFormShown(object sender, EventArgs e)
{
    Mod mod;
    RichTextboxBuilder builder;
    List<Mod> batch = task as List<Mod>;
    Refresh();
    if (batch != null)
    {
        RichTextboxBuilder.BeginConcurrentAppendProcess(this, batch.Count);
        ReportCaption = "Conversion Progress";
        progressBar.Visible = true;
        progressBar.Maximum = batch.Count;
        Parallel.For(0, batch.Count, i =>
        {
            mod = batch[i];
            builder = RichTextboxBuilder.BeginConcurrentAppend(i);
            //builder.TextUpdated += Builder_TextUpdated;
            taskTarget.ConvertToESL(mod, builder, false);
            RichTextboxBuilder.EndConcurrentAppend(i);
        });

        Finalize(false);
    }
    else
    {
        mod = task as Mod;
        ReportTextBuilder = new RichTextboxBuilder(this);
        Finalize(taskTarget.ConvertToESL(mod, ReportTextBuilder));
    }
}

Answer 1

As mentioned in the comment, I use a thread-safe HashSet

var fileQueue = new HashSet<string>(StringComparer.Ordinal);

You can use Lock() or manage it through a ReaderWriterLockSlim to make it threadsafe.

Another issue I faced is the fact that "I'm not alone on a server", meaning other processes do other stuff, shocking I know but sometimes I do not get the whole server to myself ;-)

Have a look at Nuget Package Walter

it has an extension method

TryDiscoverWhoisBlocking(this FileInfo file, out IReadOnlyList<Process> processes)

I could paste the code in here but there are quite a few native methods and the post would be too long.

Use the method to see who is blocking your access to the file, it might be a virus scanner, if so and you get the error then loop for a while allowing the virus scanner to do its thing till all handles are of the file and you can continue.

I found out that most of the time "I'm the problem" and "I'm the one blocking access" so I look at my code and figure, why use these one-liners when there actually is a way to do things the way I intended it to be done.

Bellow, I tell the type of blocking I think are needed and only take the blocking level I need. In your case, you could open shared-read and write without blocking or with... anyway the choice is yourse.

using (var fs = new FileStream(path: file.FullName, access: FileAccess.Write, mode: FileMode.Append, share: FileShare.ReadWrite))
using (var sw = new StreamWriter(fs, encoding: UTF8Encoding.UTF8))
{
    sw.Write(text);
    sw.Flush();
}

I'm writing this as I guess your code works and the locking is caused by another thread or another process, above code will definitively tell you what's going on.

Answer 2

Walter's code is definitely something that may help with other people with this problem. However my problem (and the inconsistency which I described accompanying its occurrence) were the result of Windows Messages duplicates being sent to the Message Loop. If the OnFormShown message occurred twice, it would call the message handler twice. This modification solved this particular issue:

I added a "processed" class variable and changed the handler added to my original post to the following. (The adding Task.Run was to solve a different problem).

private void OnFormShown(object sender, EventArgs e)
{
    if (!processed)
    {
        processed = true;
        Mod mod;
        RichTextboxBuilder builder;
        List<Mod> batch = task as List<Mod>;
        Refresh();
        if (batch != null)
        {
            RichTextboxBuilder.BeginConcurrentAppendProcess(this, batch.Count);
            ReportCaption = "Conversion Progress";
            progressBar.Visible = true;
            progressBar.Maximum = batch.Count;

            Task.Run(() =>
            {
                Parallel.For(0, batch.Count, i =>
                {
                    mod = batch[i];
                    builder = RichTextboxBuilder.BeginConcurrentAppend(i);
                    //builder.TextUpdated += Builder_TextUpdated;
                    taskTarget.ConvertToESL(mod, builder, false);
                    RichTextboxBuilder.EndConcurrentAppend(i);
                }); Finalize(false);
            });
        }
        else
        {
            mod = task as Mod;
            ReportTextBuilder = new RichTextboxBuilder(this);
            Finalize(taskTarget.ConvertToESL(mod, ReportTextBuilder));
        }
    }
}