How to write a large amount of random bytes to file

Question

I need to generate some files of varying sizes to test an application, I thought the easiest way to do so would be to write a Node script to do so.

I wrote the following code but the process crashes when the file size exceeds my memory.

const fs = require("fs");
const crypto = require('crypto');

const gb = 1024 * 1024 * 1024;

const data = crypto.randomBytes(gb * 5);

fs.writeFile('bytes.bin', data, (err) => {
    if (err) throw err;
    console.log('The file has been saved!');
});

Answer 1

On top of your memory issue you will also have an issue with the crypto module as the amount of bytes it can generate is limited.

You will need to use fs.createWriteStream to generate and write the data in chunks rather than generating it in one go.

Here is a modified version of some code from the Node documentation on streams to stream chunks of random bytes to a file:

const fs = require("fs");
const crypto = require('crypto');

const fileName = "random-bytes.bin";

const fileSizeInBytes = Number.parseInt(process.argv[2]) || 1000;
console.log(`Writing ${fileSizeInBytes} bytes`)

const writer = fs.createWriteStream(fileName)

writetoStream(fileSizeInBytes, () => console.log(`File created: ${fileName}`));

function writetoStream(bytesToWrite, callback) {
    const step = 1000;
    let i = bytesToWrite;
    write();
    function write() {
        let ok = true;
        do {
            const chunkSize = i > step ? step : i;
            const buffer = crypto.randomBytes(chunkSize);

            i -= chunkSize;
            if (i === 0) {
                // Last time!
                writer.write(buffer, callback);
            } else {
                // See if we should continue, or wait.
                // Don't pass the callback, because we're not done yet.
                ok = writer.write(buffer);
            }
        } while (i > 0 && ok);

        if (i > 0) {
            // Had to stop early!
            // Write some more once it drains.
            writer.once('drain', write);
        }
    }
}

There are also online tools which let you generate files of your required size with less setup. The files are also generated on your system so they don't have to be downloaded over the wire.