Perl Threads Excessive Memory Usage

Question

I have written a Perl script that runs forever as a daemon process.

my $CHILD_THREADS_CNT=5;

my  $q= new Thread::Queue();

sub processJob{
    while (defined(my $taskHandle = $q->dequeue()) ) {
        my $obj=jobProcesser->new();
        my $result=$obj->getOutput(taskHandle);
        #jobProcessor is the large modules that computes/manipulates the job
        //MySql Query to insert '$result' into DB
    }
}

#Daemon subroutine
sub daemon{
    while (1){
        my @new_request=   #'My SQL Queries that returns all  new requests from db'
        for (@new_request){
            $q->enqueue($_);
        }
        sleep 5;
    }
}
#Main thread scans the db, and, enqueue the new job in a queue.
my $scanDB=threads->create(\&daemon);

#children perform processing
my  @child=map threads->create(\&processJob), 1..CHILD_THREADS_CNT;

$scanDB->join;
$q->enqueue(undef) for 1..CHILD_THREADS_CNT;
for (@child){
    $_->join;
}

I run this script in unix(Perl v5.8.8) using nohup. The script dies after every 4-5 days. No hup does not capture the log that explain the reason for sudden death of script(main process). I anticipated memory leaks, and used top -p PID to analyse the nohup process.

As the script runs continuously and forever, the VIRT (Virtual Memory) and RES Memory sizes grow continuously. Reached somewhere around 2GB just in 10 hours.

Essentially, once the child thread finises the job(process-Job) ie. returns $result , then the memory should have been released to OS.

• Is my thread implementation memory efficient?

• What changes and optimization is required to run the script forever, without outage, or without periodic restart?

Any help is appreciated.

Answer 1

As mentioned in the comments - there's no obvious leaks here.

You've avoided some of the key 'gotchas' with thread programming by running 'worker' threads from a queue.

I would suggest you need to look at what's happening with your your database calls and your object creating in the processJob sub. Common traps in OO is to create a circular reference within an object, so even when the object goes out of scope, the ref count doesn't ever drop to zero.

It might be worth checking if a destructor is called. ( http://perldoc.perl.org/perlobj.html#Destructors )

DESTROY { 
    my ( $self ) = @_; 
    warn $self . " object destroyed";
}