Ruby File IO - Failed to allocate memory

Question

Below is a method which inserts records into the devices database. I am having a problem where I get a 'failed to allocate memory' error.

It is being run on a Windows Mobile device with quite limited memory.

There are 10 models, one is reasonably large with 108,000 records.

The error occurs when executing this line (f.readlines().each do |line|) but it occurs after the largest model has already been inserted.

Is the memory not being released by the block that is iterating through the lines? Or is there something else happening?

Any help on this matter would be greatly appreciated!

def insertRecordsIntoRhom(models)
     updateAmount = 45 / models.length

     GC.enable

     models.each_with_index do |model,i|
        csvColumns = Array.new

        db = ::Rho::RHO.get_src_db(model)
        db.start_transaction
        begin 
           j=0          

           f = File.new("#{model}.csv")
           f.readlines().each do |line|

              #extract columns from header line of csv
              if j==0
                 csvColumns = getCsvFieldFromHeader(line)
                 j+=1
                 next
              end 

              eval(models[i] + ".create(#{csvPutIntoHash(line,csvColumns)})")

           end

           f.close
           db.commit
        rescue
           db.rollback
        end
     end
 end

Answer 1

IO#readlines returns an Array, ie it reads the whole file and returns a list of all the lines. No line can be garbage collected until you are completely done iterating that list.

Since you only need one line at a time, you should use IO#each_line instead. This will read only a little bit at a time and pass you lines one by one. Once you are done with a line, it can be garbage collected while the rest of the file is being processed.

Finally, note that Ruby comes bundled with a good CSV library , you probably want to use that if you can instead of rolling your own.