NoMemoryError: failed to allocate memory - Ruby on Rails

Question

I generated a simple script that on other occasions has worked for me, but this is because the amount of information in the loop generates a NoMemoryError error.

I have 16 GB of memory and also a lot of virtual memory available. When I perform the test, the RAM memory is completely filled.

The script is:

  require 'rest-client'
  require 'json'
  require 'open-uri'
  require 'csv'

  def self.qos7705egressdiscard_summary
   xml = File.read("#{Rails.root}/public/Discard_qos7705egress.xml")
   data = RestClient.post("http://10.140.255.1:8080/xmlapi/invoke", xml,{:"Content-Type" => 'application/soap+xml'})
   data_parsed = Hash.from_xml(data)
   return data_parsed
  end

  def self.samqos7705egressdiscardtotal_table
   tabletotal = Hash.new
   data_stats = qos7705egressdiscard_summary['Envelope']['Body']['findResponse']['result']['service.SapEgrQosQueueStatsLogRecord']

   data_stats.map do |qosdiscard_device|
    @devicetotal = qosdiscard_device["monitoredObjectSiteName"]
    @servicetotal = qosdiscard_device["monitoredObjectPointer"]
    @porttotal = qosdiscard_device["displayedName"]
    @queueIdtotal = qosdiscard_device["queueId"]
    @discardinproftotal = qosdiscard_device["droppedInProfOctets"].to_i
    @discardoutproftotal = qosdiscard_device["droppedOutProfOctets"].to_i
    time_unixtotal = (qosdiscard_device["timeCaptured"]).to_i/1000
    @timeCapturedtotal = Time.at(time_unixtotal).strftime("%B %e, %Y at %I:%M %p")
    @discardtotal = @discardinproftotal + @discardoutproftotal
    @device_int_stats_total = (@devicetotal+@porttotal+@queueIdtotal).to_s
     hash = Hash[devicetotal: @devicetotal, servicetotal: @servicetotal, porttotal: @porttotal, queueIdtotal: @queueIdtotal, discardtotal: @discardtotal, device_int_stats_total: @device_int_stats_total, timeCapturedtotal: @timeCapturedtotal, time_unixtotal: time_unixtotal]
     tabletotal << hash
     #tabletotal.write(hash)
   end
  end

The exact error is:

NoMemoryError: failed to allocate memory
        from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:108:in `inspect'
        from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:108:in `block in <module:IRB>'
        from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:101:in `call'
        from /usr/local/lib/ruby/2.2.0/irb/inspector.rb:101:in `inspect_value'
        from /usr/local/lib/ruby/2.2.0/irb/context.rb:383:in `inspect_last_value'
        from /usr/local/lib/ruby/2.2.0/irb.rb:661:in `output_value'
        from /usr/local/lib/ruby/2.2.0/irb.rb:490:in `block (2 levels) in eval_input'
        from /usr/local/lib/ruby/2.2.0/irb.rb:623:in `signal_status'
        from /usr/local/lib/ruby/2.2.0/irb.rb:486:in `block in eval_input'
        from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:245:in `block (2 levels) in each_top_level_statement'
        from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:231:in `loop'
        from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:231:in `block in each_top_level_statement'
        from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:230:in `catch'
        from /usr/local/lib/ruby/2.2.0/irb/ruby-lex.rb:230:in `each_top_level_statement'
        from /usr/local/lib/ruby/2.2.0/irb.rb:485:in `eval_input'
        from /usr/local/lib/ruby/2.2.0/irb.rb:395:in `block in start'
        from /usr/local/lib/ruby/2.2.0/irb.rb:394:in `catch'
        from /usr/local/lib/ruby/2.2.0/irb.rb:394:in `start'
        from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/console.rb:110:in `start'
        from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/console.rb:9:in `start'
        from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /usr/local/lib/ruby/gems/2.2.0/gems/railties-4.2.0/lib/rails/commands.rb:17:in `<top (required)>'
        from bin/rails:4:in `require'
        from bin/rails:4:in `<main>'Maybe IRB bug!

On line 25 I added this tabletotal.write (hash) to be able to write it on disk and not in memory but I got the following error:

NoMethodError: undefined method `write' for {}:Hash

What is the problem here? Also how can I fix it?

Answer 1

Start by making it less cr*p:

def extract_data(input)
  {
    devicetotal: input["monitoredObjectSiteName"],
    servicetotal: input["monitoredObjectPointer"],
    porttotal: input["displayedName"],
    queueIdtotal: input["queueId"],
    discardinproftotal:  input["droppedInProfOctets"].to_i,
    discardoutproftotal: input["droppedOutProfOctets"].to_i,
    time_unixtotal: input["timeCaptured"].to_i/1000
  }.tap do |h|
    h[:timeCapturedtotal] = Time.at(h[:time_unixtotal]).strftime("%B %e, %Y at %I:%M %p"),
    h[:discardtotal] = h[:discardinproftotal] + h[:discardoutproftotal]
    h[:device_int_stats_total] =(h[:devicetotal]+h[:porttotal]+h[:queueIdtotal]).to_s
  end
end

This method is really easy to test since you just insert input and write assertions about the output.

If you want to map and apply this to the input array you would do:

data_stats.map(&:extract_data)

Your code tries to output a hash but uses the shovel operator like on a array. You need to decide if the appropiate output is an array or a hash.

I need all those variables because I use them in a html in table format.

This won't work - the instance variables will only contain the values from the last element as they get overwritten in each iteration.

You instead need to iterate over an array of hashes or objects in the view:

<% @stats.each do |s| %>
  <tr>
    <td><%= s[:devicetotal] %></td>
    # ...
  </tr>
<% end %>

If you really want to use instance varibles you need to create a object instance for each element:

class DataStat
  include ActiveModel::AttributeAssignment
  attr_accessor :devicetotal, :servicetotal # ...

  def self.from_import(input)
    self.new(
      devicetotal: input["monitoredObjectSiteName"],
      servicetotal: input["monitoredObjectPointer"],
      # ...
    )
  end  
end

@stats = data_stats.map { |ds| DataStat.from_import(ds) }

You also need to deal with the issue that you´re running out of memory since you are just slop converting the whole XML document into a Ruby hash. You need to parse it with Nokogiri instead and extract what you actually need.