Continuously load data (blobs) into a POST request to node.js

Question

I think this should be fairly simple, but I think I am looking into things too much and it's not making sense.

What I am currently doing

I am creating a web app using Node + React to record audio in the browser. I'm using RecordRTC on the client side to record the audio from the user's microphone. All is fine and dandy, but sometimes it takes a long time to upload the audio file after the user is finished singing. I want to process this file before sending it back to the user in the next step, so speed is critical here as they are waiting for this to occur.

In order to make the experience smoother for my users, I want to kick off the audio upload process as soon as I begin to receive the audio blobs from RecordRTC. I can get access to these blobs as RecordRTC allows me to pass a timeslice value (in ms) and an 'ondatavailable' function, that will get passed a blob every timeslice amount of milliseconds.

What I have tried

Currently I have it all easily working with FormData() as I only send the file once the user has finished singing.

1. My first idea was to find an example of something like the Fetch API being used in a manner that resembles what I'm after. There are plenty of examples, but all of them treat the source file as already being available, but as I want to continually add blobs as they come (without being able to pre-determine when these might stop coming, as a user may decide to stop the singing process early) this doesn't look promising.
2. I then considered a 'write my own' process whereby many request are made instead of attempting one long continuous style one. This would involve attaching a unique identifier to each request, and having the server concatenate each chunk together where the ids match. However, I'm not sure how flexible this would be in the future in say a multi-server environment, not to mention handling dropped connections etc, and no real way to tell the server to scrap everything in the case of a user aborted event such as closing the tab/webpage etc.
3. Finally, I looked into what was available through the likes of NPM etc without success, before conceding that perhaps my Google Fu was letting me down.

What I want

Ideally, I want to create a SINGLE new request once the recording begins, then take the blob every time I receive it in 'ondataavailable', send it to my request (which will pump it through to my server once it receives something) indefinitely. Once the audio stops (I get this event from RecordRTC as well so can control this), I want to finish/close up my request so that the server knows it can now begin to process the file. As part of the uploading process, I also need to pass in a field or two of text data in the body, so this will need to be handled as well. On the server side, each chunk should be immediately accessible once the server receives it, so that I can begin to create the audio file/append to the audio file on the server side and have it ready for processing almost immediately after the user has finished their singing.

Note: The server is currently set to look for and process multi-part uploads via the multer library on npm, but I am more than happy to change this in order to get the functionality I want.

Thanks!

Answer 1

Providing an update for anyone that may stumble upon this question in their own search.

We ended up 'rolling our own' custom uploader which, on the client side, sends the audio blobs in chunks of up to 5 1-second blobs to the server. Each request contains a 'request number' which is simply +1 of the previous request number, starting at 1. The reason for sending 5 1-second blobs is RecordRTC, at least at the time, would not capture the final X number of seconds. EG. If using 5 second blobs instead, a 38 second song would lose the final 3 seconds. Upon reaching the end of the recording, it sends a final request (marked with an additional header to let the server know it's the final request). The uploader works in a linked list style to ensure that each previous request has been processed before sending the next one.

On the server side, the 5 blobs are appended into a single 5 second audio blob via FFMPEG. This does introduce an external dependency but we were already using FFMPEG for much of our application so it was an easy decision. The produced file has the request number appended to its filename. Upon receiving the final request, we use FFMPEG again to do a final concatenation of all the received files to get our final file.

On very slow connections, we're seeing time savings upwards of 60+ seconds, so it has significantly improved the app's usability for slower internet connections.

If anyone wants the code to use for themselves please PM through here. (It's fairly unpolished but I will clean it up before sending)