Shell script memory consumption

Question

I have a ruby script that performs a lot of memory intensive work. The script sometimes fails to complete because of unavailability of memory. Fortunately, there is a way I can split the script into two as there are literally 2 parts to the ruby script. The memory intensive work also gets split into 2 as I split the script into 2 separate scripts. But now I want script 2 to execute after script 1. I intend to write a shell script which looks something like

ruby script1.rb
ruby script2.rb

What I'm concerned about in this approach is that since both scripts are scheduled in the same shell script, it would do no good to memory disintegration that we are trying to achieve by splitting the ruby script into 2.

Will both script1.rb and script2.rb run in their own memory space if they run as one shell script? And if script1.rb is terminated, would it free up memory that script2.rb might utilize? Does splitting the scripts into two and running them through a shell script sound like an approach for the memory problem?

Answer 1

(I don't really know Ruby, but I'll give it a shot)

Your case sounds as if:

• You have a Ruby script that consists of two parts, A and B

• A is executed first and uses lots of memory

• When A finishes it does not clean up after itself and leaves lots of references to useless objects

• B is executed afterwards and uses even more lots of memory

• B runs out of memory

By splitting the Ruby script into two scripts you allow the memory used by A to be implicitly freed by the operating system when it terminates. Each script is a new process and as long as they are not executed concurrently (in parallel), they do not affect each other.

Therefore, using a shell script to execute A and B consecutively allows B to work as if A had never used any memory, so it is a workaround - a very ugly one.

Since you can apparrently run A and B consecutively, you should fix A so that it cleans up after itself and frees any memory. Setting all references to any object that is not needed after A is done to nil , will allow the Ruby garbage collector to free any used memory.

The best approach would be to go through your Ruby script and set the object references to nil as soon as you are done with that object, rather than do that at the end. You may find that your script has significanty reduced memory usage afterwards.

Keeping references to unnecessary objects is a form of memory leak that can lead to a program using inordinate amounts of memory. Lists and trees that grow indefinitely, due to objects being added but never removed, are a very common cause for this. This can also significantly reduce the performance of the code, since parsing trees and - especially - lists get slower as they get bigger.

A good mental test is: "Does my algorithm need as much memory on paper?"