Netty App Design For Large Number of Small Transfers

Question

I'm trying to decide how to design a Netty app with what I think are some unusual requirements. Basically there's a client that initiates a request. That request translates into English as "Go recursively get a bunch of little tiny files under directory /whatever/, and all I can tell you about those files is that their names are between AAAAAAA.bin and CCCCCCC.bin".

So, the server needs to take the request, and start scanning some directories on the server side, and start rapidly streaming all these little files back. Performance is critical, but so is making sure that I've received all files between AAAAAAA.bin and CCCCCCC.bin.

So would it be a good design to make the client and server basically asynchronus themselves? In other words, the client initiates the conversation, sends the request, and simply receives an acknowledgement UUID token or something, and then the server begins gathering up files (maybe one per thread), contacts the client, and hands it a single file along with the UUID? I'm thinking that the client could periodically ask the server "are you done streaming my request that matched UUID token /sometoken/?

I'm not quite sure how this would be configured, since the client and server both would be initiating conversations. Or, maybe someone else has a better design idea? Again, performance (total time from request initiation to completion of all file transfers) is critical.

Thanks!

Answer 1

Assuming that you're in complete control of the protocol (ie you're not limited to HTTP), then perhaps something like

1. Client connects to server and sends directory request. If client is restarting an aborted transfer is sends a request with the token from 2
2. Server responds with a unique token for this transfer. If transfer is being restarted it responds with the token from 1.
3. Server identifies all files for this transfer, gives each file a unique id, and associates the file set with the token from 2 (might want to figure out the files before generating the token)
4. For each file the server sends a message consisting of file length, unique file id, file (and any other information, such as filename). The server sends each file ASAP and does not wait for the acknowledgement from 5.
5. Client acknowledges each file received using unique file id.
6. After last file sent, server sends a "transfer complete" message.

All of the above communication happens over a single channel. The important point is that you're streaming the files and receiving acknowledgements asynchronously thus mitigating network latency.

If you have a lot of files I wouldn't use a thread per file. Perhaps a thread pool where each file to send is added to a job queue, or perhaps each unique directory is added to the job queue and a thread processes a directory at a time. You may need to synchronise calls to channel.write(..). I'm also assuming that it's ok for the client to receive files out of order.

Actually, I would initially get it going with just one thread to read the files. Once it's working reliably look to see if having multiple threads enables you to increase performance by keeping the network busy (ie not waiting to read the next file).

When writing to the channel I'd probably write on object containing the file details (unique id, file data if small enough, filename if necessary) and then have a codec that can convert the object to / from a channel buffer.

Depending on your exact circumstances the client could open multiple connections to the server and you could assign a connection to a specific file reading thread thus avoiding any channel synchronisation issues. You may get some performance increase that way but, most likely, you'll just see the available bandwidth shared between the connections.