cURL works from Terminal, but not from PHP

Question

I'm running into a rather strange issue.

I'm trying to log into a remote moodle install using curl from PHP.

I have a curl command, which works perfectly in the Terminal.

When I translate the same thing into PHP, it works, but it just doesn't login. The exact same value which successfully login via terminal, somehow trips up the login system via PHP and it doesn't login. Instead, it returns the login page again.

My cURL command (data section ommitted as it has my username and password):

curl 'http://moodle.tsrs.org/login/index.php'
-H 'Pragma: no-cache'
-H 'Origin: http://moodle.tsrs.org'
-H 'Accept-Encoding: gzip, deflate'
-H 'Accept-Language: en-US,en;q=0.8'
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
-H 'Cache-Control: no-cache'
-H 'Referer: http://moodle.tsrs.org/login/index.php'
-H 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'
-H 'Connection: keep-alive'

The corresponding PHP code:

function login() {
    $username = $_POST['username'];
    $password = $_POST['password'];

    if(!isset($_POST['username']) || !isset($_POST['password'])) {
        echo "No login data received";
        return;
    }

    $creq = curl_init();

    $data = array('username' => $username, 'password' => $password, 'testcookies'=> '1');

    $headers = array('Pragma: no-cache', 'Origin: http://moodle.tsrs.org', 'Accept-Encoding: ', 'Accept-Language: en-US,en;q=0.8', 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36', 'Content-Type: application/x-www-form-urlencoded', 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Cache-Control: no-cache', 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'Connection: keep-alive' );
        curl_setopt_array($creq, array(
        CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_POST => true,
        CURLOPT_ENCODING => '',
        CURLINFO_HEADER_OUT => true,
        CURLOPT_POSTFIELDS => $data,
        CURLOPT_HTTPHEADER => $headers,
        CURLOPT_FOLLOWLOCATION => false
    ));

    $output = curl_exec($creq);

    echo print_r(curl_getinfo($creq));

    echo "\n" . $output . "\n";
}

And the output of curlinfo:

Array
(
    [url] => http://moodle.tsrs.org/login/index.php
    [content_type] => text/html; charset=utf-8
    [http_code] => 200
    [header_size] => 541
    [request_size] => 945
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 1.462409
    [namelookup_time] => 0.002776
    [connect_time] => 0.330766
    [pretransfer_time] => 0.330779
    [size_upload] => 365
    [size_download] => 8758
    [speed_download] => 5988
    [speed_upload] => 249
    [download_content_length] => -1
    [upload_content_length] => 365
    [starttransfer_time] => 0.694866
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [primary_ip] => 125.22.33.149
    [redirect_url] =>
    [request_header] => POST /login/index.php HTTP/1.1
Host: moodle.tsrs.org
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 365
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=----------------------------83564ee60d56


)

Does anyone know any possible reason for this? I've tried swapping out the hard coded cookie with COOKIEFILE and COOKIEJAR, but it doesn't change anything.

Answer 1

This could have been debugged better by seeing everything that was actually done by cURL. This is done by adding the verbose flag to the command: -v .

$ curl localhost/login [...] -v

We can get the same output from PHP's curl by adding the CURLOPT_VERBOSE option. Note that by adding this line you are instructing cURL to output the same information to STDOUT - it will not be returned and content will not be sent to the browser, so this must be debugged in the terminal.

curl_setopt($curl, CURLOPT_VERBOSE, 1);

By doing it this way, you can get a consistent and comparable output of both HTTP requests, it should look sommthing like this:

POST / HTTP/1.1
Host: localhost:3000
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 250
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=------------------------b4d79f17a3887f2d

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< ETag: W/"2-mZFLkyvTelC5g8XnyQrpOw"
< Date: Thu, 22 Dec 2016 19:13:40 GMT
< Connection: keep-alive

Left: Command line cURL as provided in the question (with extra -v flag)

Right: PHP cURL as posted in the question (with CURLOUT_VERBOSE enabled)

As you can see, the headers aren't the same, and this makes that clear. The PHP invocation is missing Accept-Encoding and Referer headers.

---

If that didn't turn up anything, let's try changing some more cURL settings in PHP back to the original cURL defaults.

Internally, PHP opts to override some defaults in cURL without telling you. While these settings should be fine, let's change them back by explicitly reseting them back to cURL defaults:

curl_setopt($curl, CURLOPT_DNS_CACHE_TIMEOUT, 60);
curl_setopt($curl, CURLOPT_DNS_USE_GLOBAL_CACHE, 0);
curl_setopt($curl, CURLOPT_MAXREDIRS, -1);
curl_setopt($curl, CURLOPT_NOSIGNAL, 0);

Answer 2

Use http_build_query on the $data array before passing to curl to avoid Content-Type: application/x-www-form-urlencoded; boundary=--- Content-Type: application/x-www-form-urlencoded; boundary=--- . This also ensures to encode any special characters from the password.

curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($data));

Reshape your curl requests as follows:

Make a GET request to the login page with pointing a cookie file at $cookies = '/tmp/some/dir/xyz.cookie.txt' . Make sure using full path for cookie name. And then close the curl handle. This will store the cookie in cookie file.

$creq = curl_init();
curl_setopt_array($creq, array(
  CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => '',
  CURLINFO_HEADER_OUT => true,
  CURLOPT_HTTPHEADER => $headers,
  CURLOPT_FOLLOWLOCATION => false,
  CURLOPT_COOKIEJAR => $cookies // save cookie
));
$output = curl_exec($creq);
curl_close($creq);

Now make the POST request with second curl request. This time point the same cookie file with COOKIEFILE option.

$creq = curl_init();
curl_setopt_array($creq, array(
  CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_POST => true,
  CURLOPT_ENCODING => '',
  CURLINFO_HEADER_OUT => true,
  CURLOPT_POSTFIELDS => http_build_query ($data),
  CURLOPT_HTTPHEADER => $headers,
  CURLOPT_FOLLOWLOCATION => false,
  CURLOPT_COOKIEJAR => $cookies, // save cookie
  CURLOPT_COOKIEFILE => $cookies // load cookie
);
$output = curl_exec($creq);
curl_close($creq);

It can happen sometimes the server look for the cookie when a login request made (to ensure that the request came after visiting the login page).

Answer 3

Most likely your problem is related to HTTP header Expect: 100-continue that cURL sends by default for each POST request.

The Expect: 100-continue header is used in POST requests containing big data when client is not sure that server will accept such request. In this case client first sends request with only headers including Expect: 100-continue and, if the server's response is successful, send the same request with body (POST data).

The problem is that not all web servers handle this header correctly. In such cases sending this header is undesired.

The solution is manually remove Expect header from sending headers by passing array('Expect:') to CURLOPT_HTTPHEADER option. In your case you can simply add 'Expect:' string to $headers array:

$headers[] = 'Expect:';

Answer 4

I solved the issue by setting a User-Agent

$headers = array(
        'Accept: */*',
        'User-Agent: curl/7.68.0',
        'Accept-Encoding: deflate,gzip,br',
        'Content-Type:application/json',
);

Answer 5

I suspect your first attempt using the curl command is using the GET method in the index.php file. I suggest you enable --trace-ascii on your first curl request in the command line and see whether a GET request is being made by the page or not. If yes, you should change your PHP script which is using the POST method. If you change the CURLOPT_POST to false, the PHP script should work.