Webrtc datachannels: saving data in file during transfer of big files

Question

I'm using the WebRTC data channels to build a file transfer service.

Its going quite good with smaller files, under 30 Mb or so. Right now on the receiving end I am simply saving the file data in memory, when all data is transferred I save the file.

Kinda like this :

//On the recieving side
var dataArray = [];
var dcOnMessage= function(event){
    dataArray .push(event.data);
    if(bytesToRecieve == 0)
    {
        var blob = new Blob(dataArray ,{type: incFileDesc.type});
        reader.onload = function (event) {
            saveToDisk(event.target.result,incFileDesc.name);
        }
        reader.readAsDataURL(blob);
    }
}

var saveToDisk = function(fileUrl, fileName) {
        var save = document.createElement('a');
        save.href = fileUrl;
        save.target = '_blank';
        save.download = fileName || fileUrl;
        var event = document.createEvent('Event');
        event.initEvent('click', true, true);

        save.dispatchEvent(event);
        (window.URL || window.webkitURL).revokeObjectURL(save.href);
    }

So I want to save the data on a file on disk, and then write directly to that file. But how do I do that?

Answer 1

Due to the lack of a way to append data to a blob (see the BlobBuilder API which was never implemented in all browsers) what you do is currently the best way to do it. That might change once Chrome (like Mozilla already does) supports sending blobs over the datachannel .

The filetransfer sample works reasonably well for files up to a gigabyte.

Answer 2

I don't think you can save files on disk (for security reasons), but you can save it to the indexedDB as a BLOB. IndexedDB is widely supported now (see http://caniuse.com/#search=indexeddb ) and is suited for local large objects store. See https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API for more details about the API. Here is an example for saving BLOB in IndexedDB: https://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/

Answer 3

I'm afraid the current standardized APIs don't easily allow that (see Philipp's response). The closest would be to save each as a blob/etc in localstorage/indexeddb, then use a Blob constructor to build the final file from the set of blobs. It will still have a temporary memory hit of roughly 2x filesize. Or just hold onto each blob in memory until building the final Blob and saving to disk (still a memory hit, but gradual until it goes 2x when building the final blob). These likely start having problems when the sizes of the final files get in the magnitude range of the available RAM.

Direct transfer of a single large Blob in Firefox->Firefox works today without SCTP ndata support (which isn't available yet) using a deprecated low-level chunking mechanism; it avoids the 2x part of the memory hit.

There's a non-standard API in Chrome that can mostly do the append-to-file part, last I checked. This has been an ongoing area of discussion with the WebAPI folk; it's probably time to poke them again.