PHP 或 cURL 是否以某种方式缓存其 web 响应？

Question

I have a php script making requests to some web site. I run this script from command line so no web server on my side is involved. Just pure PHP and a shell.

The response is split into pages so I need to make multiple requests to gain all the data with one script run. Obviously, the request's URL is identical except one parameter. Nothing complicated:

$base_url = '...';
$pages = ...; // a number I receive elsewhere
$delay = ...; // a delay to avoid too many requests 
$p = 0;
while ($p < $pages) {
   $url = $base_url . "&some_param=$p";
   ... // Here cURL takes it's turn because of cookies
   sleep($delay);
}

The pages I get this way look all the same - like the first one that was requested. (So I get just a repetitive list multiplied by the number of pages.)

I decided that it happens because of some caching on the web server's end which persists despite of an additional random parameter I pass. Closing and reinitializing cURL session doesn't help as well.

I also noticed that if I quickly fix the initial $p value manually (so requests start from different page) and then launch the script again, the result changes. I do it quicker than $delay value. It means that two different requests made from the same script run give same result , while two different requests made from two different script runs give different results , regardless of delay between the requests. So it can't be just caching on the responded side.

I tried to work that around and wrapped the actual request in a separate script which I run using exec() from the main script. So there is (should be, I consider) a separate shell instance for any single page request, and those requests should not share any kind of cache between them. Despite of that, I keep getting the same page again. The code looks something like that:

$pages = ...; 
$delay = ...; 
$p = 0;
$command_stub = 'php get_single_page.php';
while ($p < $pages) {
   $command = $command_stub . " $p";
   exec($command, $response);
   // $response is the same again for different $p's
   sleep($delay);
}

If I again change the starting page manually in the script, I get a result for that page all over again. Until I change it once more. And so on. Several minutes may pass between two runs of the main script, and it still yields identical result until I switch the number by hand.

I can't comprehend why this is happening. Can somebody explain it?

Answer 1

The short answer is no. Curl certainly doesn't retain anything between executions unless configured to do so (eg: setting a cookie file).

I suspect the server is expecting a session token of some sort (cookie or other HTTP header are my guess). Without the session token it will just ignore the request for subsequent pages.