Running a PHP cronjob for more than 15 minutes

Question

I am participating to the Amazon Product Advertising API and they say in their terms of conditions, that I must update my data at least hourly. Ok no problem you think, easily mass-update the 800 objects in mysql db. But there's another point: I am allowed only to make 1 Amazon query per second, so I want to timeout each query, which will then last the php execution for about 15 minutes. So this is a problem for me, because my provider configured a max execution time for php scripts of 20 seconds and I cannot change it.

So I thought of http-redirecting the php script to itself. Again no problem till now. But now's the problem: I want to htaccess-password-protect the script, my provider of cronjobs can handle htaccess pw protection. Without any protection this would be to unsafe, since there could be some heavy DB traffic if the script is abused.

Question: Is it possible to login to the htaccess login dialog with php, since there is no human beeing that can do the login process? Or do you have any other idea to solve this issue? Thank you.

Answer 1

Adressing your initial problem: execution time

There are two places you can set this.

1. configuration - which you can't access or change apparently

2. runtime - I don't know any possibility to disallow changing the max execution time on runtime

In your script add this to the top. set_time_limit will set your execution time to unlimited, all other integer values are in seconds

<?php
set_time_limit(0);

Answer 2

Adressing your second question : providing http auth credentials programmatically

This is just for the interested reader, I don't think (hope for you) you'll need this for that case

I'll send raw HTTP so this can be used for every kind of client and language which is able to open a tcp socket connection

My source: Basic access authentication

<?php
$username = 'myuser';
$password = 'mypass';

// computing authorization digest
$httpAuthDigest = base64_encode( "{$username}:{$password}" );

// connecting to the remote server
$fp = fsockopen( 'www.zappos.com', 80, $errno, $errstr, 60 );
if (!$fp) {
    die( "{$errstr} ({$errno})\n" );
}

// send headers
fputs( $fp, "HEAD /womens-clothing HTTP/1.1\r\n" );
fputs( $fp, "Host: www.zappos.com\r\n" );
fputs( $fp, "Authorization: Basic {$httpAuthDigest}\r\n" );
fputs( $fp, "Connection: close\r\n" );
fputs( $fp, "\r\n" );

// receive result ( optional )
while (!feof($fp)) {
    echo fgets($fp, 128);
}
fclose($fp);

This returned the most exciting response header I know :)

HTTP/1.1 200 OK
Server: nginx/1.1.17
Content-Type: text/html; charset=utf-8
X-ZFC-Metadata: KjYIExIJCgNuaWQSAjQ1EhYKBmxheW91dBIMdGhyZWVfY29sdW1uEg8KA3pjMRIIQ2xvdGhpbmc=
X-Powered-By: Ponies!
X-Varnish-TTL: 60m
X-Varnish: 248743314 248742997
X-Cache-Hits: 4
X-Varnish-Host: varnish04.zappos.net
X-Varnish-ID: drupal
X-Core-Value: 5. Pursue Growth and Learning
X-Recruiting: If you're reading this, maybe you should be working at Zappos instead.  Check out jobs.zappos.com
X-UUID: 9dc901f6-38ca-11e2-be29-00145e157f03
Cache-Control: max-age=3085
Date: Tue, 27 Nov 2012 19:50:38 GMT
Connection: close

Answer 3

OP stated he was still getting a timeout after a specific time, although overwriting the configured max_execution_time to unlimited, which was working according to phpinfo()

Consider it to be bad practise to call a worker script via a browser. This can cause uncontrollable states and causes other troubles just like those you're experiencing right now - asides from possible troubles this could cause when somebody else gets the URL into his hands.

Also - a hourly import is quite uncomfortable to do manually in a browser isn't it?

Your problem is, that not php stops execution because of a timeout, but the webserver. You said you dont pass any GET or POST parameters. Just run it on php-cli on your command line

php yourscript.php

All output will be logged into console. if you want you can write it into a logfile like this

php yourscript.php > log.txt

This still doesn't do anything automatically. On any unix you can easily set up a cronjob doing this job for you at a desired time and interval

eg

crontab -e

to edit your configured cronjobs

* */2 * * * /usr/bin/php /yourpath/yourscript.php > ~/yourlog.txt 2>/dev/null

This would run every two hours and log your script output to ~/yourlog.txt , redirect errors into /dev/null

Check this to find out how the schedule table works http://www.adminschoice.com/crontab-quick-reference

I also advice to move your script out of your webserver's document root so it is no longer accessible from the web.