PHP通过curl获取数据并插入SQL数据库

Question

I have a few questions about my code for creating a local replica of the client database via CURL GET.

Situation:

• I need to create local replica databases with 1.5mil records via curl
• Url address is generating a script. Url addresses consist of parameters SINCE, LASTID, ACCESS_TOKEN like https://CLIENT_SERVER_URL/sync?last_id=LAST_ID&since=ISO_8601_TIMESTAMP&access_token=TOKEN
• One curl call generates 100 records
• Access to the client server is limited to 60 accesses per minute
• Script to be run every day and synchronize databases

What I have problem with and I need to advise

• Q1.) BFU question (sorry), How do I generate timestamp in this format 2016-06-22T23%3A34%3A20.169659Z ? Timestamp must generate the CURRENT_DATE - 1 day

• Q2.) How do I create a loop to script repeatedly generate new URLs and store the records in the database?

• Q3.) How can I limit calls to only 60 calls per minute?

• Q4.) How do I make a INSERT into the database so that the entry is inserted when it does not exist or UPDATEd when it exists?

My PHP code

<?php 
// select DB
$servername = "MY_DATABASE";
$username = "USER";
$password = "PASS";
$dbname = "DB_NAME";

// Create connection
$corporateBodies = new mysqli($servername, $username, $password, $dbname);

$sql = "SELECT id FROM my_table ORDER BY id DESC LIMIT 1";
$lastID = $corporateBodies->query($sql);
$lastID = mysqli_fetch_array($lastID);

$base = "CLIENT_SERVER_URL/";
$since = "GENERATED_TIMESTAMP";
$accessToken = "ACCESS_TOKEN";
$url = $base . "sync?last_id=" . $lastID["id"] . "&since=" . $since . "&access_token=" . $accessToken;

//function httpGet($url) {
    $crl = curl_init($url); 
    curl_setopt($crl,CURLOPT_URL,$url);
    curl_setopt($crl,CURLOPT_RETURNTRANSFER,true);
    curl_setopt($crl,CURLOPT_HEADER, false);
    curl_setopt($crl, CURLOPT_TIMEOUT, 60); 
    $reply =curl_exec($crl);

    curl_close($crl);

    //decoding the json data
    $decoded_data = json_decode($reply, true);

    $insertArray = $corporateBodies->prepare(
    "INSERT INTO my_table (
        id,
            cin,
            tin,
            vatin,
            name,
            formatted_address,
            street,
            reg_number,
            building_number,
            postal_code,
            municipality,
            country,
            established_on,
            terminated_on,
            vatin_paragraph,
            registration_office,
            registration_number,
            formatted_street,
            street_number,
            created_at,
            updated_at
    )
    VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
    );

    $corporateBodies->query("START TRANSACTION");

    foreach ($decoded_data as $row) {
    $bind = $insertArray->bind_param('iiissssisisssssssssss',
            $row['id'],
            $row['cin'],
            $row['tin'],
            $row['vatin'],
            $row['name'],
            $row['formatted_address'],
            $row['street'],
            $row['reg_number'],
            $row['building_number'],
            $row['postal_code'],
            $row['municipality'],
            $row['country'],
            $row['established_on'],
            $row['terminated_on'],
            $row['vatin_paragraph'],
            $row['registration_office'],
            $row['registration_number'],
            $row['formatted_street'],
            $row['street_number'],
            $row['created_at'],
            $row['updated_at']
    );

    $exec = $insertArray->execute();

    // Close the prepared statement
    $insertArray->close();

    $commit = $corporateBodies->query("COMMIT");

    $corporateBodies->close();
?>

Answer 1

your timestamp perfectly matches an ISO8601 with Z instead of +00:00 , and microsecond precision, so to create your timestamp - 1 day, would probably be

$stamp=(new DateTime('-1 day',new DateTimeZone("UTC")))->format('Y-m-d\TH:i:s.u\Z');

quote How do I create a loop to script repeatedly generate new URLs and store the records in the database? if you're wondering how to create a loop,

for($i=0;$i<60;++$i){/*code in here will be looped 60 times!*/}

if you're wondering to to generate the new URLs, check the API docs, we can't answer that question without knowing how the API works, and you didn't include the api specs in your question.

if you're wondering how to store something in a database, that depends on your database, we can't answer that question without knowing what kind of database you're using. but assuming you're using an SQL database (as opposed to a NoSQL database like MongoDB, or some other database), its usually through an insert statement, like

INSERT INTO records VALUES('new record');

quote How can I limit calls to only 60 calls per minute?

while(!$finished){
$quota_reset_time=microtime(true)+60;
for($i=0;$i<60;++$i){
//code in here runs 60 times
}
@time_sleep_until($quota_reset_time); // this line sleeps until 60 seconds has passed since $quota_reset_time was set.
}

the code inside the for loop would run at a maximum speed of 60 times per minute.

quote Q4.) How do I make a INSERT into the database so that the entry is inserted when it does not exist or UPDATEd when it exists?

again, it depends on the database. assuming a MySQL or MariaDB database, though, make the value an UNIQUE column, and make an INSERT OR REPLACE statement, like

INSERT OR REPLACE INTO tbl(`value`,`last_update_time`)
VALUES('FOO',NOW());