php-将发布的JSON对象存储在数据库中

Question

I'm reading a large json file with Java and post each line of my localhost, which then takes the JSON and reads it as an object, then stores parts of the object in that database using MySQL.

This is a really slow process.

How can I optimize it?

<?php
error_reporting(0);
@ini_set('display_errors', 0);

$json = $_POST['data'];

if(!empty($json)){

    $obj = json_decode($json);

    $user_id =  $obj->interaction->author->id;
    $user_link =  $obj->interaction->author->link;
    $name =  $obj->interaction->author->name;
    $user_name =  $obj->interaction->author->username;
    $user_gender =  $obj->demographic->gender;
    $user_language =  $obj->twitter->lang;
    $user_image =  $obj->interaction->author->avatar;
    $user_klout =  $obj->klout->score;
    $user_confidence =  $obj->language->confidence;
    $user_desc =  $obj->twitter->user->description;
    $user_timezone =  $obj->twitter->user->time_zone;
    $user_tweet_count = $obj->twitter->user->statuses_count;
    $user_followers_count = $obj->twitter->user->followers_count;
    $user_friends_count = $obj->twitter->user->friends_count;
    $user_location = $obj->twitter->user->location;
    $user_created_at = $obj->twitter->user->created_at;

    $tweet_id =  $obj->twitter->id;
    $tweet_text =  $obj->interaction->content;
    $tweet_link =  $obj->interaction->link;

    $tweet_created_at =  $obj->interaction->created_at;

    $tweet_location =  $obj->twitter->user->location;

    //$tweet_geo_lat =  $obj->interaction->geo->latitude;
    //$tweet_geo_long =  $obj->interaction->geo->longitude;

    $con = mysqli_connect("localhost","root","", "cohort");
    // Check connection
    if (mysqli_connect_errno()) {
      echo "Failed to connect to MySQL: " . mysqli_connect_error();
    }

    $sql = "INSERT INTO tweeters (user_id, screen_name, name, profile_image_url, location, url,
            description, created_at, followers_count,
            friends_count, statuses_count, time_zone,
            last_update, klout, confidence, gender
    )
    VALUES ('$user_id', '$user_name','$name',
    '$user_image', '$user_location', '$user_link',
    '$user_desc', '$user_created_at', '$user_followers_count',
    '$user_friends_count', '$user_tweet_count', '$user_timezone',
    '', '$user_klout', '$user_confidence', '$user_gender' )";

    if (!mysqli_query($con,$sql)) {
      //die('Error: ' . mysqli_error($con));
    }

    $sql = "INSERT INTO search_tweets (tweet_id, tweet_text, created_at_date,
        created_at_time, location, geo_lat,
        geo_long, user_id, is_rt)
        VALUES ('$tweet_id', '$tweet_text','',
                '$tweet_created_at', '$tweet_location', '',
                '', '$user_id', '')";

    if (!mysqli_query($con,$sql)) {
      //die('Error: ' . mysqli_error($con));
    }

    mysqli_close($con);
    echo json_encode(array("id" => $user_id ));
}

?>

Java:

    String inputfile = "D:\\Datasift\\Tweets.json"; //  Source File Name.  
        double nol = 200000; //  No. of lines to be split and saved in each output file.  
        File file = new File(inputfile);  
        Scanner scanner = new Scanner(file);  
        int count = 0;  
        System.out.println("Storing file in stack"); 
        int may = 0, june = 0, just_june=0, july = 0;
        BufferedReader br = null;

        BufferedReader  in = new BufferedReader (new InputStreamReader (new ReverseLineInputStream(file)));


        while(true) {
            String line = in.readLine();
            if (line == null) {
                break;
            }
            //System.out.println("X:" + line);
            // Send POST output.
            URL url1;
        URLConnection   urlConn;
        DataOutputStream    printout;
        DataInputStream     input;
                // URL of CGI-Bin script.
        url1 = new URL ("http://localhost/json/");
                // URL connection channel.
        urlConn = url1.openConnection();
                // Let the run-time system (RTS) know that we want input.
        urlConn.setDoInput (true);
                // Let the RTS know that we want to do output.
        urlConn.setDoOutput (true);
                // No caching, we want the real thing.
        urlConn.setUseCaches (false);
                // Specify the content type.
        urlConn.setRequestProperty
        ("Content-Type", "application/x-www-form-urlencoded");
            printout = new DataOutputStream (urlConn.getOutputStream ());
            String content =
                "data=" + URLEncoder.encode (line);
            printout.writeBytes (content);
            printout.flush ();
            printout.close ();
            // Get response data.
            input = new DataInputStream (urlConn.getInputStream ());
            String str;
            while (null != ((str = input.readLine()))){
                //System.out.println (str);                   
            }

            input.close ();
        }
        System.out.println("Lines in the file: " + count);

Answer 1

我不想以任何方式进行拖钓，但是为什么不只使用PHP读取文件呢？

Answer 2

If you are repeating this process multiple times one way of doing it is using multiple data sets.

So rather than doing INSERT INTO table (field,field) VALUES (value,value) and looping it many times you would do $ins = "INSERT INTO table (field, field) VALUES"; then within your foreach loop (or each time you call the code) build up an array $ins_values[] = "(escaped_value,escaped_value)"; and then run the query $ins.implode(',',$ins_values).

Mysql will run much faster in this context although beware that Mysql sets data limits on max_allowed_packet which might need adjusting depending.

Hope that helps and I've understood your question correctly.