.NET Parallel file creation

Question

I'm creating a program that fetches file data from the database. The requirement is that the files get numbered on existence 'IMAGE_00001.JPG', 'IMAGE_00002.JPG', ...

Because of performance issues i'm loading the data parallel but I'm getting an issue with creating the files.

For i As Long = 1 To maxSeqNr
    DIDir = New DirectoryInfo(currDir)
    If DIDir.GetFiles(currFilePrefix & Microsoft.VisualBasic.Right(maxSeqNrLeadingZeros & CStr(i), Len(maxSeqNr.ToString)) & ".*", SearchOption.TopDirectoryOnly).Length = 0 Then
        sRetValue = currFilePrefix & Microsoft.VisualBasic.Right(maxSeqNrLeadingZeros & CStr(i), Len(maxSeqNr.ToString)) & currFileExtention

        Dim oSW As StreamWriter
        oSW = New StreamWriter(currDir & sRetValue)
        oSW.WriteLine("")
        oSW.Close()
        oSW.Dispose()
        oSW = Nothing

        Exit For
    End If
Next

The issue is that sometimes 2 threads are running the same line of code. I added logging and I found this:

Check for 'IMAGE_00001.JPG' ; Directory: ''
Check for 'IMAGE_00001.JPG' ; Directory: ''
Check for 'IMAGE_00002.JPG' ; Directory: 'IMAGE_00001.JPG'
Check for 'IMAGE_00003.JPG' ; Directory: 'IMAGE_00001.JPG,IMAGE_00002.JPG'

Is there a possibility to "lock" the working directory before I execute the Exists statement. And "release" after I created the file? The other threads should wait until they can "lock" the directory.

Answer 1

You should generate the name that you want to use and then attempt to create a FileStream object using that name. Use a suitable constructor that accepts a FileMode parameter (eg this one ) and specify CreateNew .

Catch an IOException if one is thrown and generate a new name.

Otherwise, you're dealing with a classic form of race. It's up to you whether you choose to combine your original strategy (check for existence, then try to open the file) or just attempt to create the file and expect exceptions.

Once you've successfully opened your FileStream , you can pass it to a StreamWriter constructor that accepts a Stream parameter, and proceed from there.

Answer 2

I would use a syn / lock strategy: Lock when your threads wants to write a file and halt all other threads until that is done, then release the lock. http://msdn.microsoft.com/de-de/library/vstudio/ms173179.aspx

Alternatively, use a non-blocking integer as a counter, so it will be guaranteed a unique file, but it MAY mess up your "numbering" requirement. http://msdn.microsoft.com/en-us/library/system.threading.interlocked.aspx

Answer 3

If all you need to do is generate blank files IMG_00000.jpg thru IMG_XXXXX.jpg , you could use a Parallel.For loop to create the files if they do not exist. This will ensure two numbers are not checked by two different threads, instead the numbers will be carved out between threads.

Although I do not know that this will actually be any faster, especially if you must search the directory for a file with any extension matching the name of the image:

Dim width = Len(maxSeqNr.ToString)
Dim DIDir = New DirectoryInfo(currDir)
Parallel.For(1L, maxSeqNr, Sub(i)
    Dim pattern = currFilePrefix & Microsoft.VisualBasic.Right(maxSeqNrLeadingZeros & CStr(i), Len(maxSeqNr.ToString)) & ".*"
    If Not DDir.EnumerateFiles(pattern, SearchOption.TopDirectoryOnly).Any Then
        Dim fn = currFilePrefix & Microsoft.VisualBasic.Right(maxSeqNrLeadingZeros & CStr(i), width) & currFileExtention
        File.Create(fn).Close
    End If
End Sub)

If the folder you're searching in is small-ish, this may be better (assuming you actually care about any file name which matches the pattern):

' My VB.Net is horrible
Dim width = Len(maxSeqNr.ToString)
Dim files = Directory.GetFiles(currDir)
Parallel.For(1L, maxSeqNr, Sub(i)
    Dim pattern = currFilePrefix & Microsoft.VisualBasic.Right(maxSeqNrLeadingZeros & CStr(i), Len(maxSeqNr.ToString))
    ' consider case-insenstive matching
    If Not files.Any(Function(fn) Path.GetFileNameWithoutExtension(fn) = pattern) Then
        Dim fn = currFilePrefix & Microsoft.VisualBasic.Right(maxSeqNrLeadingZeros & CStr(i), width) & currFileExtention
        File.Create(fn).Close
    End If
End Sub)

Or, if all you really need is to only create a file if it does not exist, you just go ahead and touch them all:

Parallel.For(1L, maxSeqNr, Sub(i)
    Dim fn = currFilePrefix & Microsoft.VisualBasic.Right(maxSeqNrLeadingZeros & CStr(i), width) & currFileExtention
    File.Open(fn, FileMode.OpenOrCreate).Close
End Sub)