How do I write directly to a USB drive with over 4GB data?

Question

I have a research project where I need to be able to fill a USB stick with a known pattern of data directly, no files or file system. My goal is to fill a drive from top to bottom with my pattern; I have written a program for writing to /dev/sd* but it is slow and does not work if the drive is over 4GB in size. The writing will stop at offset oxFFFFFFF or 2^32.

My code

#include <stdio.h>
#include <fcntl.h>
#include <linux/fs.h>
#include <errno.h>
#include <inttypes.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/ioctl.h>
#include <ctype.h>



int main(int argc, char **argv)
{
    int fd;
    uint64_t numbytes=0;
    char response[4];
    long int nblocks = 0; // number of blocks up to 4GB


    int error;
    size_t fill_length = 8;
    char buf[512];
    char fill[] = "ABCDEFGH";

    printf("\n\n");
    fd = open(argv[1], O_RDWR);
    if(error = ioctl(fd, BLKGETSIZE64, &numbytes) != 0){
        printf("Failed to read block device, ioctl returned: %i\n", error);
        exit(0);
    }
    if (numbytes > 8589934592){ // Exit if more than 8 GB drive
        printf("Drive is too large.l\n");
        exit(0);
    }
    printf("Number of bytes: %lu, i.e. %.3f GiB\n", numbytes,
    (double)numbytes / (1024 * 1024 * 1024));
    nblocks = numbytes / 512;
    printf("Number of blocks on device: %lu\n", nblocks);

    strcpy(buf, fill); // fills with pattern, one time
    for(int i =0; i < (512 - fill_length); i += fill_length){ // Fills up the rest of the buffer
        strcat(buf, fill); // with the pattern to be repeated.
    } // 512 is the default & smallest block size
    printf("buf is:\n%s\n", buf);

    printf("\n*** The device at %s will be completely overwritten ***\n", argv[1]);
    printf("\nAre you sure you want to proceed? (Type:<Ctrl>-C to exit)\n");

    // printf("\n nblocks: %lu", nblocks);
    fgets(response, 3, stdin);
    printf("writting to: %s\n", argv[1]);

    for (int i = 0; i <= nblocks; i++)
    {
        write(fd, buf, 512);
    }


    printf("Closing...\n");
    close(fd);
    printf("Closed.\n");
    return 0;

} 

I realize my program isn't great and is dangerous as I could wipe out a HDD, but all I am looking for at this point is tips to make this work on drives over 4GB and hopefully make the process faster. It will have limited use by myself and possibly another.

A push in the right direction would be appreciated.

Answer 1

Use size_t for bytesize in memory, and off_t for file offsets on disk. For general integers, use intptr_t in your program. So your for loops should start with for (intptr_t i=0;

And don't write in small blocks of 512 bytes, but in something bigger (but a power of two), eg 16384 bytes.

If your program still does not work, use strace to find out the failing syscalls. And use perror in your program on failure of any syscall.

Answer 2

your code is 'effectively' writing to a hard disk that is larger than a 32bit int.

Therefore, I suggest paying attention to the built-in USB interface on the USB stick, where you can specify which logical block to write and so forth.

Note; USB sticks with built-in load levelling will NOT write sequentially on the media, irregardless of what you pass to it in the way of commands. However, most USB sticks can be told to format and so forth so you might be able to use that feature. Or you could use some utility for setting disk sectors to all (for instance) 0's Just be carefully to not overwrite the sector/block formatting information.

Answer 3

You can make off_t into a 64 bit value by placing this before any of the includes.

#define _FILE_OFFSET_BITS 64    // so off_t is 64 bit, see man fseeko

#include <stdio.h>
#include <unistd.h>
etc. ...

This also apparently also causes the read() , write() , fread() , fwrite() , etc. calls manage a 64-bit file offset internally. I have used it for a similar purpose when porting to Linux.

Ditto the advice not to use such a small buffer. Make it 64K (65,536 bytes) to greatly improve performance.