Ruby, Puma & Sinatra: display streaming output

Question

I have a Puma app that, when you enter an IP address and choose to traceroute it by clicking the checkbox, it will perform a traceroute :

Code below (the app in reality does more but for the sake of this question I have simplified it) :

Site.erb:

class Pumatra < Sinatra::Base

    get '/' do
        erb :index
    end

    post '/run' do
    params.to_s
    end

 get'/traceroute_results' do
        @ipaddress  = params[:ipaddress]
        @traceroute = params[:traceroute]

            if @traceroute == "on"
                    stdout, status = Open3.capture2("traceroute -4 -w3 #{@ipaddress}")
                    @traceroute_result ="<pre><code>" + stdout + "</code></pre>"
            end

       @traceroute_header
            @traceroute_result
        end

        @traceroute_thread = Thread.new{traceroute()}
        @traceroute_thread.join

        erb :traceroute_results
    end
end

And my views/index.erb file contains this (showing you guys only the relevant bits):

....
        <div class="checkbox">
            <label><input type="checkbox" id="traceroute" 
             name="traceroute">Traceroute</label>
        </div>
....

        <!-- Start Results block -->
        <section id="results_block" style="display:none;" class="l_panel bg_color_white l_relative">
            <div id="traceroute_results" style="display:none;" class="l_grid"></div>
        </section>
        <!-- End Results block -->

And this all works fine. Traceroute runs and the result is shown once it completes. However, sometimes this takes ages to complete due to hops being unresponsive, although this is normal and it does give you the result back eventually. My question is: for cosmetic reason only, I would like to display the output traceroute as it's being run. So, more of a progressive view, than a completed output in the end. Is this possible and how would I go about doing this?

Appreciate the hints, J

EDIT:

I tried this in site.erb

   get'/traceroute_results' do
                @traceroute_header = "<br>TCP Traceroute results:"
                  IO.popen("traceroute -4 -w3 #{@ipaddress}") do |io|
                        io.each do |line|
                        @traceroute_result = line
                        erb :traceroute_results
                        end
                  end

    end

But now my app doesnt show any output for traceroute :(

My traceroute_results.erb file :

<%= @traceroute_header %>
<%= @traceroute_result %>

EDIT: Amadan's helpful post has led me to this code in my site.erb:

 get'/traceroute_results' do
        @ipaddress  = params[:ipaddress]
        @traceroute = params[:traceroute]

          if @traceroute == "on"
            ipaddress  = params[:ipaddress]
              stream do |res|
                res << erb(:traceroute_header)
                Open3.popen3("traceroute -T -4 #{@ipaddress}") do |stdin, stdout, 
                  stderr, wait_thr|
                  while line = stderr.gets
                    res << erb(:traceroute_results, {}, { out: line })
                  end
                end
              end
          end
  end

And changed my traceroute_results file to :

<%= out %>

Unfortunately I am still receiving nothing when the url loads. I can tell its performing the traceroute because it takes awhile before the result comes back, but it comes back empty.

Really really appreciate anyone's help and patience in this. I'm running Ruby Puma with Sinatra. Can anyone run this successfully in such a setup? I havent been able to find examples with Sinatra that actually works in the puma context... :(

Thanks, J

Answer 1

Firstly, note that erb does not send a response; it normally becomes the response because its result is the last thing in a Ruby function, which gets returned implicitly, and Sinatra responds with the get block's return value. Your current implementation renders your subtemplate but discards each of the results; meanwhile, IO.popen(...) {...} returns nil , which is the cause of no response.

Now, in order to generate a streaming response in Sinatra, you can return something that responds to #each , or use the stream helper to create such an object for you. Try to change into something like this:

get '/traceroute_results' do
  ipaddress  = params[:ipaddress]
  stream do |res|
    res << erb(:traceroute_header)
    IO.popen("traceroute -4 -w3 #{ipaddress}") do |io|
      io.each do |line|
        res << erb(:traceroute_result, {}, { line: line })
      end
    end
  end
end

__END__
@@ traceroute_result
<div><%= line %></div>

@@ traceroute_header
<h4>Traceroute results</h4>

Answer 2

Amadan's answer is both correct and incomplete, in a way that could cause thread starvation on the server when using each .

I'm not a Sinatra developer, but I write server code and this is why I feel compelled to warn against the use of each .

The issue with each is that the server has no control of the thread while waiting for each to iterate and return - this blocks the thread from handling other requests .

A better solution would use a thread pool and a queue for calls to traceroute and an SSE / WebSockets connection for streaming output from the queue to the user.

This would protect the server from DoS situations and allow it to:

1. Show a "waiting" massage (perhaps with the queue's length) when too many requests arrive at the same time.

2. Stream data from the traceroute to the client as it becomes available without blocking the server (pushing data to the client).

3. Cancel a traceroute if a client disconnects.

This can be performed using WebSocket native solutions or using Hijacking solutions.