如何使用 socket.io 和大于 1mb 的 nodejs 实现房间到房间的文件共享系统？

Question

I want to implement socket.io room to room file sharing system so users can send images to their respective rooms to all user can see it and I have tried using base64 encoding method to emit sender image file to specific room but it can send only approximately 700kb to 800kb file.

Is there any easier way of doing this and can support larger files above 1mb and it should be able to load images progressively?

I am using ejs template engine, nodejs, socket.io, javascript.

Console.log("please help me guys if you any idea about this, I tried many things but none of them are working and I have read the socket.io documentation but didn't get any clue about it

I have also tried binary streaming but got no luck please help me guys with some codes samples

Answer 1

You will probably find it easier for the client to upload the file to your http server with a room name and then have your http server send a message to all the other clients in the room via socket.io with a URL where the client can download the file using http. socket.io is just not a streaming protocol, it's a packet or message-based protocol so to send large things, it has to be broken up into messages and then reassembled on the client. This can be done, but it's just extra non-standard work that http uploads and downloads already know how to do.

Here would be the steps:

1. Client uploads file to server via http post with the room name as a field in the form.
2. Server receives uploaded file, assigns it a unique ID and stores it in a temporary location on the server on disk.
3. When file upload completes server notifies all other clients in the room via socket.io that the file is uploaded and ready for download and sends them the URL for download that has the uniqueID in it.
4. Each client sends request to download the file via http using the unique URL they received.
5. Server serves the file to each client as requested over http.
6. Server either keeps track of whether all clients have now finished downloading or just removes the file after some period of time based on timestamp of the file (to just clean up disk space) with some regular cleanup function on a recurring timer.

You can create a single route that handles all the downloads:

const downloadRoot = "/temp/filexfer";

app.get("/download/:id", (req, res) => {
    const fullPath = path.resolve(path.join(downloadRoot, req.params.id));
    // detect any leading . or any double .. that might jump outside
    // the downloadRoot and get to other parts of the server
    if (!fullPath.startsWith(downloadRoot)) {
         console.log(`Unsafe download request ${fullPath}`);
         res.sendStatus(500);
         return;
    } 
    res.download(fullPath);
});

A cleanup algorithm could look like this:

const fsp = require('fs').promises;
const path = require('path');
const oneHour = 1000 * 60 * 60;

// run cleanup once per hour
let cleanupTimer = setInterval(async () => {
    let oneHourOld = Date.now() - oneHour;
    try {
        let files = await fsp.readdir(downloadRoot, {withFileTypes: true});
        for (let f of files) {
             if (f.isFile()) {
                 let fullName = path.join(downloadRoot, f.name);
                 let info = await fsp.stat(fullName);
                 // if file modification time is older than one hour, remove it
                 if (info.mtimeMs <= oneHourOld) {
                     fsp.unlink(fullName).catch(err => {
                         // log error, but continue
                         console.log(`Can't remove temp download file ${fullName}`, err);
                     });
                 }
             }
        }
    } catch(e) {
        console.log(e);
    }
    
}, oneHour);

// unref the timer so it doesn't stop node.js from exiting naturally
cleanupTimer.unref();

Answer 2

There are a lot of ways to do this sort of thing and it depends heavily on what sort of architecture you want to support.

Sending large files through socket.io or any other web-socket is fine. it does require a bunch of chopping and reassembling on your web app but it will work.

WebRTC is another way to share files of any description, and it will not burden your server in anyway which is good. (here is a tutorial on it https://ably.com/tutorials/web-rtc-file-transfer )

The issue with either of these methods is that they are transient shares, are new user to the room will not get the image, unless your server re-transmits the data again.

My suggestion would be to upload the file to s3 directly and then share a link to it that can be resolved on each of the clients. This will keep the server burden down and reduce your storage requirements in the backend server