PHP Curl and headers problem

Question

I have asked a question on this before, but again ran into problems and just cannot solve it. I have a internal proxy server and a content server. The code on the proxy server is this. (Some comments may be wrong but leaving here to convey what is my understanding):

<?php
session_start();
$data_server_url = "http://my_data_server_url/";
$i_var_prefix="i_var_";

$process_headers_separately=0;
//$process_headers_separately=1;
// WARNING! Has problems with GZIPPED DATA!
// AVOID/REMOVE OPTION ALLTOGETHER
// (Set to 1 if you want to catch received headers
// and send explicit headers to clients)
//-----------------------------------------

// Other important request dependent 'SERVER' variables.
if(isset($_SERVER['HTTPS']))
{ $_POST["${i_var_prefix}_HTTPS"]=$_SERVER['HTTPS']; };

if(isset($_SERVER['REMOTE_ADDR']))
{ $_POST["${i_var_prefix}_REMOTE_ADDR"]=$_SERVER['REMOTE_ADDR']; };

$request_uri="";
if(isset($_SERVER['REQUEST_URI'])) { $request_uri = $_SERVER['REQUEST_URI']; };
$curl_url="${data_server_url}${request_uri}";

$field_array= array(
      'Accept' => 'HTTP_ACCEPT',
      'Accept-Charset' => 'HTTP_ACCEPT_CHARSET',
      'Accept-Encoding' => 'HTTP_ACCEPT_ENCODING',
      'Accept-Language' => 'HTTP_ACCEPT_LANGUAGE',
      'Connection' => 'HTTP_CONNECTION',
      'Host' => 'HTTP_HOST',
      'Referer' => 'HTTP_REFERER',
      'User-Agent' => 'HTTP_USER_AGENT'
      );

$curl_request_headers=array();

foreach ($field_array as $key => $value) {
   if(isset($_SERVER["$value"])) {
      $server_value=$_SERVER["$value"];
      $curl_request_headers[]="$key: $server_value";
   }
};
//------
session_write_close();

//Open connection
$curl_handle = curl_init();
curl_setopt($curl_handle,CURLOPT_COOKIE,session_name()."=".session_id().";");
//Set the url, number of POST vars, POST data
curl_setopt($curl_handle, CURLOPT_URL, $curl_url);
curl_setopt($curl_handle, CURLOPT_POST, count($_POST));
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $_POST);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl_handle, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($curl_handle, CURLOPT_HEADER, $process_headers_separately);
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $curl_request_headers);
curl_setopt($curl_handle, CURLOPT_ENCODING, "identity");


//Execute post
$result = curl_exec($curl_handle);

//Close connection
curl_close($curl_handle);

if ($process_headers_separately) {
   list($headers,$content)=explode("\r\n\r\n",$result,2);
   foreach (explode("\r\n",$headers) as $hdr) {
      header($hdr);
   }
   echo $content;
} else {
   echo $result;
}    
?>

Problem 1: With the current code, even if the Content-Type returned by the data_server is text/plain, the content-type seen by the client is text/html. For example, see http://sarcastic-quotes.com/robots.txt This request goes to the file above. I have checked that the data server is actually returning Content-Type as text/plain. But through the proxy, client sees content-type in response headers as text/html.

Problem 2: Note the use of variable process_headers_separately . If I set it to 1, then the browser tries to download a gzip file instead of displaying the contents (no matter what content-type the data server returns). Thus, there is some logical bug in that code flow.

I just want the above code to work as an internal proxy that seamlessly acts as a bridge between my data server and the client. Any thoughts would be appreciated, I am really confused with how to correctly handle the headers above.

regards, JP

---

I have found the cause of the problem when process_headers_separately=1 (browser downloading file instead of displaying). But it is SO strange and am unable to solve it. Problem: If I uncomment the lines if(isset($_SERVER['REMOTE_ADDR'])) { $_POST["${i_var_prefix}_REMOTE_ADDR"]=$_SERVER['REMOTE_ADDR']; }; if(isset($_SERVER['REMOTE_ADDR'])) { $_POST["${i_var_prefix}_REMOTE_ADDR"]=$_SERVER['REMOTE_ADDR']; }; then things start working fine! Strange!

Its gotta be some crazy whitespace problem like Neil suggested.

Anyways, trying to fix this - I have spent 3 days (2 earlier and 1 now) due to this crazy bug :(. Thanks to Neil and RF for helping me on this.

Answer 1

This all looks pretty kosher. The only discrepancy I can see is that SOME servers only separate headers with \\n , rather than \\r\\n . But that's probably not the issue.

Can you try writing var_export(explode("\\r\\n",$headers),true) to a file and see what comes out?

Re your gzip issue, make sure there's no trailing whitespace after your closing ?> , or even safer, change echo $result to die($result) . Obviously if you don't pass on the CURL response's headers, Apache will make up its own, so you're going to need that CURLOPT_HEADER flag.

Is CURL decompressing the response for you? If so, you may need to intervene with the Content-Encoding and Content-Length response headers.

n.

Answer 2

You could use the following lines of code:

//Execute post
$result = curl_exec($curl_handle);
$result_info = curl_getinfo($curl_handle);

And then:

} else {
    if (isset($result_info['content_type'])) {
        header("Content-Type: {$result_info['content_type']}");
    }
    echo $result;
}

Basically, your proxy server is the one that's returning the data. So, that's where you would have to set the Content-Type. By default, this would be html. But if you check if the CURL response had a specific Content-Type and manage to set that, it'll do the trick! :)

Let us know if that works for you.

---

I kinda understand what you're trying to do now. So here's how to get the headers in your proxy's response:

If you set the CURLOPT_HEADER option to 1, you will receive the headers in your output which will let you do this:

// Grab the headers from the result
if ($process_headers_separately) {
    list($header_string, $content) = explode("\r\n\r\n", $result, 2);
    // OR :
    // $header_string = substr($result, 0, $result_info['header_size']);
    $headers = http_parse_headers($header_string);
    foreach ($headers as $key => $value) {
        header("$key: $value");
    }
}

I suggest using the http_parse_headers function because of the reasons mentioned by the poster of the first answer. I'm not sure if this will solve your gzipping problem. Let us know.