PHP cURL set encoding?

Question

I use cURL to transfer data between two servers - both running Ubuntu 12.04, Lighttpd and PHP5.5 FastCGI. I used to bzcompress the data - this is purely a legacy issue: I had found that bzcompress gives more efficient compression when writing out text data to files. The data transferred tended to be quite small - typically of the order of under 512 bytes.

However, today I ran into an issue when the data were somewhat longer - closer to 1 kB. curl_exec duly returned true and no errors were reported. However, the data never arrived at their destination. My original code was as follows

curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,1);
curl_setopt($ch,CURLOPT_POSTFIELDS,"{$cql}");
curl_setopt($ch,CURLOPT_RETURNTRANSFER,false);

I suspected that there was an issue with encoding and, perhaps with my bzcompression. So I replaced bzcompress with gzdeflate and altered my curl code to

curl_setopt($ch,CURLOPT_HTTPHEADER,array('Content-Type:text/plain'));
curl_setopt($ch,CURLOPT_ENCODING,'');
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,1);
curl_setopt($ch,CURLOPT_POSTFIELDS,"{$cql}");

This works - no more vanishing data. However, the "solution" is purely the result of cobbling together an alternative based on some reading of the PHP documentation and some of the posts here - not a thing to be relied on for what is a small but mission critical part of what I am doing.

So the question - what is going on here? Why did the original code fail with longer data strings and why does the latter version work? Will it always work or is there something else that is missing?

I would much appreciate any help and tips.

Answer 1

Time to answer my own question. The "solution" I have outlined above is little more than a red herring. It will not get you anywhere so don't bother trying it.

The real issue here is best appreciated by noting that the server in question is Lighttpd. Having researched matters a bit I have discovered that it has an unfortunate habit of sending out an HTTP 417 header when it gets a request with a http Expect: 100-continue header.

Either it appears to do this in an intermittent manner or else libcurl appends this header in an intermittent manner - perhaps when POST data length exceeds a certain threshold.

In any case this was at the root of my problems. The right solution is to edit the lighttpd.conf file, add a

server.reject-expect-100-with-417 = "disable"

line, and reload the server configuration

sudo service lighttpd force-reload

This prevents your own lighty installation from falling over when an Expect 100-Continue is encountered but does little if a libcurl request you make reaches a lighty (or another) server which throws up its hands in dismay when it sees that. The only way to avoid that

curl_setopt($ch,CURLOPT_HTTPHEADER,array("Expect:  "));

This has cost me half a day of work. Hopefully, it will help someone else here.