How do I use cURL/PHP to dump the current twitter trending topics into mySQL

Question

Trying to run a cron job every few hours that reads the current Twitter Trending topics listed here:

http://search.twitter.com/trends.json

And then dump the top 10 trends into a mySQL table on my server

how to do this? thanks

Answer 1

Here a couple of pointers that could help you :

To use curl : you will need to initialize a connexion, configure it, and execute it.

For instance, something like this might do (very basic example) :

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://search.twitter.com/trends.json");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$json = curl_exec($ch);

curl_close($ch);

// $json contains the data you want
var_dump($json);

You will get that kind of output :

string '{"as_of":"Fri, 21 Aug 2009 20:59:54 +0000","trends":[{"name":"Follow Friday","url":"http:\/\/search.twitter.com\/search?q=%22Follow+Friday%22"},{"name":"#BrutalLegend","url":"http:\/\/search.twitter.com\/search?q=%23BrutalLegend"},{"name":"#shoutout","url":"http:\/\/search.twitter.com\/search?q=%23shoutout"},{"name":"#fact","url":"http:\/\/search.twitter.com\/search?q=%23fact"},{"name":"Inglourious","url":"http:\/\/search.twitter.com\/search?q=Inglourious"},{"name":"Which Horror Movie","url":"http:\/\/search.twitter.com\/search?q=%22Which+Horror+Movie%22"},{"name":"Cataclysm","url":"http:\/\/search.twitter.com\/search?q=Cataclysm"},{"name":"Inglourious Basterds","url":"http:\/\/search.twitter.com\/search?q=%22Inglourious+Basterds%22+OR+%22Inglorious+Basterds%22"},{"name":"District 9","url":"http:\/\/search.twitter.com\/search?q=%22District+9%22"},{"name":"Twitter Besides","url":"http:\/\/search.twitter.com\/search?q=%22Twitter+Besides%22"}]}' (length=955)

Of course, you might want to set a couple of other options ; for the complete list, take a look at the documentation of curl_setopt .

Parsing the JSON string :

Provided you are using PHP >= 5.2, you can use json_decode to parse the json data :

$data = json_decode($json);

var_dump($data);

You will then get something like this :

object(stdClass)[1]
  public 'as_of' => string 'Fri, 21 Aug 2009 21:01:48 +0000' (length=31)
  public 'trends' => 
    array
      0 => 
        object(stdClass)[2]
          public 'name' => string 'Follow Friday' (length=13)
          public 'url' => string 'http://search.twitter.com/search?q=%22Follow+Friday%22' (length=54)
      1 => 
        object(stdClass)[3]
          public 'name' => string '#BrutalLegend' (length=13)
          public 'url' => string 'http://search.twitter.com/search?q=%23BrutalLegend' (length=50)
      2 => 
        object(stdClass)[4]
          public 'name' => string '#shoutout' (length=9)
          public 'url' => string 'http://search.twitter.com/search?q=%23shoutout' (length=46)
      3 => 
        object(stdClass)[5]
          public 'name' => string '#fact' (length=5)
          public 'url' => string 'http://search.twitter.com/search?q=%23fact' (length=42)
      4 => 
        object(stdClass)[6]
          public 'name' => string 'Inglourious' (length=11)
          public 'url' => string 'http://search.twitter.com/search?q=Inglourious' (length=46)
      5 => 
        object(stdClass)[7]
          public 'name' => string 'Cataclysm' (length=9)
          public 'url' => string 'http://search.twitter.com/search?q=Cataclysm' (length=44)
      6 => 
        object(stdClass)[8]
          public 'name' => string 'Which Horror Movie' (length=18)
          public 'url' => string 'http://search.twitter.com/search?q=%22Which+Horror+Movie%22' (length=59)
      7 => 
        object(stdClass)[9]
          public 'name' => string 'Inglourious Basterds' (length=20)
          public 'url' => string 'http://search.twitter.com/search?q=%22Inglourious+Basterds%22+OR+%22Inglorious+Basterds%22' (length=90)
      8 => 
        object(stdClass)[10]
          public 'name' => string 'District 9' (length=10)
          public 'url' => string 'http://search.twitter.com/search?q=%22District+9%22' (length=51)
      9 => 
        object(stdClass)[11]
          public 'name' => string 'Hurricane Bill' (length=14)
          public 'url' => string 'http://search.twitter.com/search?q=%22Hurricane+Bill%22' (length=55)

This contains the whole data obtained from twitter.

Inserting the data into database :

Now, you have to iterate over this array of 'trends', and, for each line, insert it into your MySQL database.

For this, you can use either :

• mysqli
• or PDO

Using prepared statements might help, too ( mysqli , pdo ) ;-)
If you are not using prepared statements, anyway, you must thing about escaping your data, using either mysqli_real_escape_string or PDO::quote .

Here, you will of course need to already have a table, with the right structure ; I also suppose you know how to insert data in a MySQL table.

If you have more specific questions, don't hesitate to ask, with some examples of code you are using that doesn't work (and a description of what doesn't work / the error message you are getting, of course) !

Have fun !

Answer 2

<?php


$curl = curl_init();
curl_setopt_array($curl, Array(
    /* CURLOPT_HEADER         => true,
    CURLOPT_POSTFIELDS     => $postdata,
    CURLOPT_POST           => $method == 'POST',
    CURLOPT_HTTPHEADER     => $headers,
    CURLOPT_USERAGENT      => "Mozil.....", */
    CURLOPT_URL            => "http://search.twitter.com/trends.json",
    CURLOPT_TIMEOUT        => 300,
    CURLOPT_CONNECTTIMEOUT => 60,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_SSL_VERIFYHOST => false,
    CURLOPT_SSL_VERIFYPEER => false,
    CURLOPT_ENCODING       => 'gzip,deflate'
));
$json = curl_exec($curl);
$json = json_decode($json);
echo "<pre>";
print_r($json);


?>

Have fun.