Rackspace, load balancing, flush(), and cURL

Question

First off, I apologize if this question is vague or confusing. The truth is, I'm pretty confused about what's going on, and I'm even more confused about how to remedy it.

I am working on a Rackspace cloud server. As you hopefully know (if you are reading this question) Rackspace employs a load balancer which tends to kill scripts that last over 30 seconds. Unfortunately, I am working on a huge processing script that involves the transfer of many large files. Needless to say, the script can take as much as an hour to run sometimes. Fortunately, this script doesn't need to be run too often. Unfortunately, I am stuck with the Rackspace cloud—for my boss, upgrading is simply not an option.

By using PHP's flush() function, coupled with some menial whitespace, I am able to avoid this timeout issue nearly always. It's not a great solution, but given my limitations and the use of my transfer script, it works pretty well.

...when I load the script in my browser.

When I try to call the script with a cron job, the script just times out. It times out when I try to call it from Hurl.it as well. However, when I load it in the browser, everything seems to work again.

How can I remedy this issue? I need to call this script using a cron job, but no matter what I do, it seems to always fail.

Thanks for the help!

Answer 1

Depends on what is timing out the script.

If it's the racksace load balancer timing out, you are able to adjust the load balancer timeout through the API up to 120 seconds. If you need the script to run longer than that, you will probably need to split up the work into several smaller sections. This is a good idea anyway because a long execution time invites problems of network dropouts where several smaller scripts can be run and each validated more easily.

Ref:

• http://docs.rackspace.com/loadbalancers/api/v1.0/clb-devguide/content/Update_Load_Balancer_Attributes-d1e1812.html

=== If you're hitting the php.ini max_execution_time, then calling set_time_limit(20) each iteration through a loop (I don't know anything about your code) or similar, will continually reset the timer.

From php.net:

When called, set_time_limit() restarts the timeout counter from zero. In other words, if the timeout is the default 30 seconds, and 25 seconds into script execution a call such as set_time_limit(20) is made, the script will run for a total of 45 seconds before timing out.

Ref:

• http://www.php.net/manual/en/function.set-time-limit.php