使用cUrl登录到ThinkStock

Question

I'm wondering whether it is possible to cUrl into ThinkStock. My probing/research so far has yielded the following:

• The auth page https://secure.thinkstockphotos.com/Authentication/SignIn sets required cookies - Try clearing domain cookies after visiting that page, then log-in, it will keep you on that page.

• If I try and get those cookies in one request without sending login POST data, upon the next request it will ask for a CAPTCHA.

I'm not sure what to do at this point, I have tried matching headers as best as possible, using referers, FF4 user agent, but it doesn't seem enough. I was wondering if anybody has run into this before with ThinkStock.

Disclaimer: This is not a scraper, it is to build up a database of images from our previous downloads for easier access to our agency's designers/not unnecessarily decreasing our download limit for that day.

My code for initial connection:

function __construct(
    $username = THINKSTOCK_USERNAME, $password = THINKSTOCK_PASSWORD){

    // create the cookie file
    $this->cookieFile = tempnam('./cookies', '');
    //******** Connect to host and try and get cookies required for auth *******//
    // initialise the CURL connection
    $curlRequest = curl_init('https://secure.thinkstockphotos.com/Authentication/SignIn');

    // do not attempt to verify the SSL certificate
    curl_setopt($curlRequest, CURLOPT_SSL_VERIFYPEER, false);

    // set curl to follow up to two redirects
    curl_setopt($curlRequest, CURLOPT_FOLLOWLOCATION, true);
    // curl_setopt($curlRequest, CURLOPT_MAXREDIRS, 5);
    curl_setopt($curlRequest, CURLOPT_REFERER, 'https://secure.thinkstockphotos.com/Authentication/SignIn');
    // set curl to timeout after two seconds
    curl_setopt($curlRequest, CURLOPT_CONNECTTIMEOUT, 5);
    curl_setopt($curlRequest, CURLOPT_TIMEOUT, 5);

    // set the cookie file
    curl_setopt($curlRequest, CURLOPT_COOKIEJAR, $this->cookieFile);

    // set the POST data
    curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1');


    // do not output the returned data
    curl_setopt($curlRequest, CURLOPT_RETURNTRANSFER, true);

    // execute the curl request and close the connection
    $response = curl_exec($curlRequest);
    var_export($response);
    curl_close($curlRequest);


    //******** Connect to host with cookies and gogogo auth *******//
    // initialise the CURL connection
    $curlRequest = curl_init('https://secure.thinkstockphotos.com/Authentication/SignIn');

    // do not attempt to verify the SSL certificate
    curl_setopt($curlRequest, CURLOPT_SSL_VERIFYPEER, false);

    // set curl to follow up to two redirects
    curl_setopt($curlRequest, CURLOPT_FOLLOWLOCATION, true);
   // curl_setopt($curlRequest, CURLOPT_MAXREDIRS, 5);
    curl_setopt($curlRequest, CURLOPT_REFERER, 'https://secure.thinkstockphotos.com/Authentication/SignIn');
    // set curl to timeout after two seconds
    curl_setopt($curlRequest, CURLOPT_CONNECTTIMEOUT, 5);
    curl_setopt($curlRequest, CURLOPT_TIMEOUT, 5);

    // set the cookie file
    curl_setopt($curlRequest, CURLOPT_COOKIEFILE, $this->cookieFile);
    curl_setopt($curlRequest, CURLOPT_COOKIEJAR, $this->cookieFile);

    // set the POST data
    curl_setopt($curlRequest, CURLOPT_POST, true);
    curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1');


    // do not output the returned data
    curl_setopt($curlRequest, CURLOPT_RETURNTRANSFER, true);


    curl_setopt(
        $curlRequest,
        CURLOPT_POSTFIELDS,
        array(
          'userName' => $username,
          'password' => $password,
          'returnUrl' => '/',
          'SignInButton' => ''
        ));

    // execute the curl request and close the connection
    $response = curl_exec($curlRequest);
    curl_close($curlRequest);
    var_export($response);
    // if the log in attempt failed, throw an exception
    if (strpos($response, 'https://secure.thinkstockphotos.com/Authentication/SignIn') !== false){
      throw new Exception('Incorrect log-in details');
    }

}

Answer 1

ThinkStock's terms of service prohibit what you appear to be trying to do. Even if you have rights to the images since you've paid to download them etc, they do not allow site access via automation. I think you need to contact them for permission, and then ask if there is an alternative login page you can use. Given the nature of what you want to do, they may relent (since you're not trying to rip off the site, just make it easier to access the content you have licensed)

Use of the Site

This Site and the Thinkstock Content are intended for customers of Thinkstock. You may not use this Site or the Thinkstock Content for any purpose not related to your business with Thinkstock. You are specifically prohibited from: (a) downloading, copying, or re-transmitting any or all of the Site or the Thinkstock Content without, or in violation of, a written license or agreement with Thinkstock; (b) using any data mining, robots or similar data gathering or extraction methods; (c) manipulating or otherwise displaying the Site or the Thinkstock Content by using framing or similar navigational technology; (d) registering, subscribing, unsubscribing, or attempting to register, subscribe, or unsubscribe any party for any Thinkstock product or service if you are not expressly authorized by such party to do so; and (e) using the Site or the Thinkstock Content other than for its intended purpose. Such unauthorized use may also violate applicable laws including without limitation copyright and trademark laws, the laws of privacy and publicity, and applicable communications regulations and statutes.