Streaming files from EventMachine handler?

Question

I am creating a streaming eventmachine server. I'm concerned about avoiding blocking IO or doing anything else to muck up the event loop.

From what I've read, ruby's non-blocking IO can be used to stream files in a non-blocking way, or I can call next_tick, but I'm a little unclear about which of these approaches is preferable.

Part of the problem is that I have not found a good explanation of non-blocking IO library functions in ruby.

Short version : Assuming a long-lived network IO operation, several wall clock minutes of streaming per file, transfer, what is the best way to do this in eventmachine without gumming up the event loop?

  while 1 do
    file.read do |bytes|
      @conn.send_data bytes
    end
  end

I understand that the above code will block and I'm wondering what to put in its place. Also, I cannot use the FileStreamer class that is part of eventmachine as is, because I need to manipulate the data after it's read but before it's sent.

Answer 1

I think you can still use FileStreamer. FileStreamer expects its first argument to be a Connection, but this is a loose contract. As long as you implement the methods that FileStreamer expects, it should work. Take a look at this

https://gist.github.com/f4d997c3eeb6bdc5a9f3

The methods you'll need to handle are send_data and send_file_data . You can perform your manipulations here. Then pass the result along to EM::Connection.

Also, from my reading of the code, the special property of FileStreamer is that it allocates a memory mapped file (unless the file is small). You could do essentially the same thing by opening a regular Ruby File, reading blocks out of it, doing your manipulation, and emulating the behavior of FileStreamer.stream_one_chunk . Which is basically:

• Each iteration must either send some data to the Connection, or reschedule itself using next_tick
• Data can be repeatedly written to the Connection until the outbound buffer is full (according to get_outbound_data_size )
• Once the file has been fully read, it should be closed (of course)

In fact, it seems to me that you had better not use FileStreamer unless your file will comfortably fit in memory.

You can look at the EM::Protocols for ideas about how to transform the data as it is streaming through.