stackoom
  简体   繁体   中英

GET Request works with python 'requests' library, but does not work with curl

 原文  2015-08-23 20:38:09  4   1    php/ python/ curl

Question

I need to probably modify my cURL request to make it work. You can find the needed url from searching on Qantas for example (Melbourne to Sydney, one way, the next day flights), you get a page with the results. Then if you mouseover the flight number, you get a dialogue box. In FF network, you can see a get request happen as you mouseover which retrieves a json document.

The cURL code that does not work: 

$flight_data_url = 'http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=eFVNx9ORnACoaRyRp_Iu675LrmM6JfRkDHpa2c6I60bhJABPBeW3!764143094!1637886517!1440118985617?AIRLINE_CODE=QF&B_DATE=201508220600&B_LOCATION=MEL&E_LOCATION=SYD&FLIGHT_NUMBER=400&LANGUAGE=GB&SITE=QFQFQFBD'

$handle = curl_init();
$options = array(
                CURLOPT_RETURNTRANSFER => true,
                CURLOPT_FOLLOWLOCATION => true,
                CURLOPT_CAINFO => $path . '\cacert.pem',
                CURLOPT_URL => $flight_data_url,
                CURLOPT_HTTPHEADER => array('Host'=>'book.qantas.com.au','User-Agent'=>'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)','Accept'=>'application/json, text/javascript, */*; q=0.01','Accept-Language'=>'en-US,en;q=0.5','Accept-Encoding'=>'gzip, deflate','Content-Type'=>'text/javascript; charset=utf-8','X-Requested-With'=>'XMLHttpRequest','Referer'=>'http://book.qantas.com.au/pl/QFOnline/wds/OverrideServlet','Cookie'=>$flight_cookie_file,'Connection'=>'keep-alive')
            );

            curl_setopt_array($handle, $options);
            $flight_model_data = curl_exec($handle);

            curl_close($handle);

The python code that does work: 

#!/usr/bin/env python
import requests


def getaurl():

    url = 'http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=LuJYL8OJHk_DUIPcYbcfT343gYMEw3b9ej4Vt_MATLerQG0kNIZR!932397328!976639289!1440293569417?AIRLINE_CODE=QF&B_DATE=201508240630&B_LOCATION=MEL&E_LOCATION=SYD&FLIGHT_NUMBER=404&LANGUAGE=GB&SITE=QFQFQFBD'

    headers = { "Host":"book.qantas.com.au",'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0', "Accept":"application/json, text/javascript, */*; q=0.01", "Accept-Language":"en-US,en;q=0.5", "Accept-Encoding":"gzip, deflate","Content-Type":"text/javascript; charset=utf-8", "X-Requested-With":"XMLHttpRequest", "Referer":"http://book.qantas.com.au/pl/QFOnline/wds/OverrideServlet", "Connection":"keep-alive" }
    r = requests.get(url, headers=headers)

    print r.text

getaurl()

1 answers

solution1
 1 ACCPTED  2015-08-23 23:55:27

Ok i debugged your code. You are doing two things wrong to get it to work.

1 ) you need to tell curl that it needs to handle a GZIP encoded connection 

curl_setopt($handle, CURLOPT_ENCODING, "gzip");

2 ) set your headers using as per manual (it is not a key => value list) search CURLOPT_HTTPHEADER in http://www.php.net/manual/en/function.curl-setopt.php

A full working example. Just change the url and the cookie value to make sure the session is working 

<?php header('Content-Type: text/html; charset=utf-8'); ?>

<html>
<head>
    <title>Fixed by Bizmate.biz</title>
</head>
<body>
<?php

$flight_data_url = "http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=Wzpc3MHW1Z1dQVnKNgjL1zGOFYs5jTi0yey2JJB3OsumjMe42a8C!-111476454!1015406667!1440372015574?AIRLINE_CODE=QF&B_DATE=201508261625&B_LOCATION=YVR&E_LOCATION=LAX&FLIGHT_NUMBER=3352&LANGUAGE=GB&SITE=QFQFQFBW";

//$flight_data_url = 'http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=eFVNx9ORnACoaRyRp_Iu675LrmM6JfRkDHpa2c6I60bhJABPBeW3!764143094!1637886517!1440118985617?AIRLINE_CODE=QF&B_DATE=201508220600&B_LOCATION=MEL&E_LOCATION=SYD&FLIGHT_NUMBER=400&LANGUAGE=GB&SITE=QFQFQFBD';

$handle = curl_init();

$flight_cookie_file = 'DWM_XSITECODE=QFQFQFBW; wt_21=false; QSession=D884F2710404EC6F60E402761C3296A8; TLTUID=8B4FCC6D7E9CD0AC141750417B7DDB37; qantas_isDevice=type#desktop|os#Linux; _wt.user-676736=WT3vpi-xU8uREVQbLyHDUf1TwZLPluq_yfjSdWFE4rxld2gVwcxlwtwsrUqhlQL-MMM-zC-j0Nk1MDugbvMEcuRZWkUrevpXCBQnAHuAQ~~; amdfa=done; pastSearchCookie=REGION:am:TRIPTYPE:O:FROM:Vancouver:TO:Sydney:DEPDAY:Wednesday:DEPMONTH:8:DEPYEAR:2015:RETDAY:Wednesday:RETMONTH:10:RETYEAR:2015:FLEXIBLE:false:ADULTS:1:CHILDREN:0:INFANTS:0:TRAVELCLASS:ECO|REGION:am:TRIPTYPE:O:FROM:Vancouver:TO:Sydney:DEPDAY:Wednesday:DEPMONTH:8:DEPYEAR:2015:RETDAY:Wednesday:RETMONTH:10:RETYEAR:2015:FLEXIBLE:true:ADULTS:1:CHILDREN:0:INFANTS:0:TRAVELCLASS:ECO|REGION:am:TRIPTYPE:R:FROM:Vancouver:TO:Sydney:DEPDAY:Monday:DEPMONTH:9:DEPYEAR:2015:RETDAY:Wednesday:RETMONTH:10:RETYEAR:2015:FLEXIBLE:false:ADULTS:1:CHILDREN:0:INFANTS:0:TRAVELCLASS:ECO|; aam_uuid=09862152334998442092616937494749808238; s_cc=true; s_fid=77B7E8432AD3E6A0-170B4CF864D887BC; s_qntttc=1440372017198; s_sq=%5B%5BB%5D%5D; s_vi=[CS]v1|2AED21DE051D0C8B-40001905C00064CC[CE]; aam_uuid=09862152334998442092616937494749808238; _wt.mode-676736=WT3PVZU041xU6Y~; webserver=server44; 3_uc=region#AM|tvldate#20150826|arr#SYD|locale#en|dep#YVR|country#ca';

$headerArray = array(
    'Host: book.qantas.com.au',
    'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
    'Accept: application/json, text/javascript, */*; q=0.01',
    'Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,it-IT;q=0.4,it;q=0.2',
    'Accept-Encoding: gzip, deflate, sdch',
    'Content-Type: text/javascript; charset=utf-8',
    'X-Requested-With: XMLHttpRequest',
    'Referer: http://book.qantas.com.au/pl/QFOnline/wds/OverrideServlet',
    'Cookie: ' . $flight_cookie_file,
    'Connection: keep-alive',
    'Pragma: no-cache'
);

$options = array(
    CURLOPT_RETURNTRANSFER => true,
    CURLINFO_HEADER_OUT => true,
    CURLOPT_ENCODING => 'gzip',
    CURLOPT_URL => $flight_data_url,
    CURLOPT_HTTPHEADER => $headerArray

);

#
curl_setopt_array($handle, $options);

$flight_model_data = curl_exec($handle);



if(curl_exec($handle) === false)
{
    echo 'Curl error: ' . curl_error($handle);
}
else
{
    echo 'Char encoding ' . mb_detect_encoding($flight_model_data) . ' <br/><br/>';

    echo 'Operation Completed without any errors : <br/>' . $flight_model_data . " ENDED\n\n <br/><br/>";

    file_put_contents("test.txt",$flight_model_data);

    echo "<br/>header : " . json_encode(curl_getinfo ($handle) );
}

curl_close($handle);
?>
</body>
</html>

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question cURL works, but an identical request in Apache's HttpComponents library does not? GET request works with CURL but not as a URL PHP curl requests with 404, curl request itself works fine CURL headers from PHP to python requests library? cURL POST request works, Guzzle 6 does not GET curl request only works in postman but not in php PHP - GET request works in Postman, but not with cUrl (PHP) curl_request_async does not work How to replicate cURL and python's 'requests' library in PHP? cUrl PHP Get request returning 400 BAD REQUEST but works in Postman
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM