Optimizing speed of read/write function on hard disk using multithreading

Question

I am trying to read -> encrypt/decrypt -> write raw data on volumes sector by sector. I need to optimize the time somehow because it is taking more than 2 hours for a volume of mere 2GB (my volume G: here). I tried using threads to handle the time problem but it had no significant results. The program needs to run on a everyday use PC.

Each Thread operates on a different block of sectors from the other threads.

Things I know:

1. Read cannot be done while writing (So I handled the critical section problem)
2. Read/Write in threads decreases time because of the seek time etc.
3. Too many threads might fail to accomplish the task due to the latency ( caused by the read/write head moving back and forth. (gives error DEVICE_NOT_READY))

Want I want to ask:

1. Can I utilize threads to write-read or write-write on two different sectors simultaneously (because I know that the disk operations are carried out using scheduling algorithms answered here )?
2. Why program generates the error when while during write in one thread (on one sector), the other threads tries to read (on another sector)?
3. What modifications would you suggest to bring down the time?

Code:

DWORD WINAPI EncryptSectorBlock(LPVOID lpParam)
{
    PTARGS args = (PTARGS)lpParam;
    static unsigned char buffer[SECTOR_SIZE];

    printf("Thread No:%i start:%i  end:%i\n ", args->thread_id, args->sector_start, args->sector_end);

    for (int i = args->sector_start; i <= args->sector_end; i++)
    {
        //Entering critical section of the code. The Other Threads Would be first spin with 65536 loops and then set to
        // sleep until the worker threads releases the lock on critical section

        if (ReadSector(args->read, buffer, SECTOR_SIZE, i) != 1)
            printf("Thread: %i. Error reading sector %i\n",args->thread_id, i);
        else
            printf("Thread: %i. Read Sector %i Successfully\n", args->thread_id, i);

        xts_encrypt_sector(buffer, i, SECTOR_SIZE, &(args->ctx));

        //Critical Section starts
        EnterCriticalSection(&CriticalSection);
        if (WriteSector(args->write, buffer, SECTOR_SIZE, i) != 1)
            printf("Thread: %i. Error writing sector %i\n",args->thread_id ,i);
        else
            printf("Thread: %i. Wrote Sector %i Successfully\n", args->thread_id, i);
        //Critical Section Ends
        LeaveCriticalSection(&CriticalSection);


        //init to zero every time in case a sector is failed to be read or write, so it can recover from fault 
        //This may break in future.. Not proper mechanism to recover from   fault. just a hack.
         memset(buffer, 0, SECTOR_SIZE);
    }
    return 0;
  }

INT_RETURN EncryptFullVolume(wchar_t* volume, unsigned char* key)
{   
       //init variables here

            for (int i = 0; i < MAX_Threads; i++)
        {
            ArgsDataArray[i] = (PTARGS)HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof(TARGS));

            if (ArgsDataArray[i] == NULL)
            {
                // If the array allocation fails, the system is out of memory
                // so there is no point in trying to print an error message.
                // Just terminate execution.
                ret = EXIT_FAILURE;
            }
            else
            {
                // Generate unique data for each thread to work with here

               // Create the thread to begin execution on its own.
                hThreadArray[i] = CreateThread(
                    NULL,                   // default security attributes
                    0,                      // use default stack size  
                    EncryptSectorBlock,     // thread function name
                    ArgsDataArray[i],          // argument to thread function 
                    0,                      // use default creation flags 
                    &ThreadIdArray[i]);   // returns the thread identifier 

                if (hThreadArray[i] == NULL)
                {
                    ret = EXIT_FAILURE;
                }

                sector_offset += sectors_per_thread;
            }

        } // End of main thread creation loop.

            // Wait until all threads have terminated.

        DWORD result = WaitForMultipleObjects(MAX_Threads, hThreadArray, TRUE, INFINITE);


        // Free all the dynamically allocated structures
    }
    }
   return ret;

}

Answer 1

Your point 2 just isn't true.

For a regular magnetic disk, the best access pattern are big sequential reads, without having to skip back and forth due to multiple threads contending the disk time (making it waste time in seeks); also, sequential reads play well with the read-ahead caching done by the operating system and the disk itself.

You may want multithreading just to process the data you read from disk, but given that on a modern PC your processor is way faster than the disk for such a task, your task is going to be IO bound. If I were you I'd just have a single thread doing asynchronous sequential IO in big chunks (think 4 MB of data a time), encrypting the data while waiting for the disk to do its thing. Another possibility is to do essentially the same thing with two threads (one doing the IO, one encrypting) and synchronous IO.