Parsing JSON data and inserting to MySQL

Question

So basically I want to parse a JSON file in PHP and insert the data into specific tables/columns. At the moment I have a working script but requires me to modify the JSON largely until it works. However, it won't end up working because the JSON data I'm collecting can vary in size having more data rows.

The JSON file is structured differently to most I have seen. Maybe because its output data from sensor units. I want to insert the data and the serial number into the data table, and have an error_log table where I can store the serial number and error messages as strings. How can I achieve this?

JSON File:

{
   "device": {
      "sn": 5165654,
      "name": "FDI_AWS_DEMO",
      "v": "2.7B3"
   },
   "channels": [
      {
         "code": "RH",
         "name": "Relative Humidity",
         "unit": "%"
      },
      {
         "code": "AT",
         "name": "Air Temperature",
         "unit": "C"
      },
      {
         "code": "MINVi",
         "name": "Min voltage",
         "unit": "V"
      },
      {
         "code": "PTi",
         "name": "Processor temperature",
         "unit": "C"
      },
      {
         "code": "SDB",
         "name": "Network signal dB",
         "unit": "dB"
      },
      {
         "code": "LWS",
         "name": "Leaf Wetness",
         "unit": "%"
      },
      {
         "code": "WSAV",
         "name": "Wind Speed Avg",
         "unit": "km/h"
      },
      {
         "code": "WSMX",
         "name": "Wind Speed Max",
         "unit": "km/h"
      },
      {
         "code": "WSMN",
         "name": "Wind Speed Min",
         "unit": "km/h"
      },
      {
         "code": "PR_TOT",
         "name": "PR Tot",
         "unit": "mm"
      },
      {
         "code": "RAIN",
         "name": "Rain",
         "unit": "mm"
      },
      {
         "code": "FDI",
         "name": "fdi",
         "unit": "Unit"
      },
      {
         "code": "DT",
         "name": "Delta-T",
         "unit": "C"
      },
      {
         "code": "LAT",
         "name": "Latitude",
         "unit": "deg"
      },
      {
         "code": "LON",
         "name": "Longitude",
         "unit": "deg"
      },
      {
         "code": "WD",
         "name": "Wind Direction",
         "unit": "Degrees"
      },
      {
         "code": "P1",
         "name": "Par1",
         "unit": ""
      },
      {
         "code": "AVGCi",
         "name": "Average Current",
         "unit": "mA"
      },
      {}
   ],
   "data": [
      {
         "$ts": 170801164400,
         "$msg": "SD_FAIL;1"
      },
      {
         "$ts": 170801170000,
         "$msg": "WDT;WV01"
      },
      {
         "$ts": 170801170000,
         "$msg": "WDT;SDI12"
      },
      {
         "$ts": 170801170000,
         "$msg": "WDT;LWS"
      },
      {
         "$ts": 170801170000,
         "RH": 67.15,
         "AT": 12.87,
         "MINVi": 3.81,
         "PTi": 23.4,
         "LWS": "0*T",
         "WSAV": 0,
         "WSMX": 0,
         "WSMN": 0,
         "PR_TOT": 156,
         "RAIN": 0,
         "FDI": 0.239,
         "DT": 2.881,
         "WD": "0*T",
         "P1": "0*T",
         "AVGCi": 175
      },
      {}
   ]
}

PHP Code:

<?php
    //connect to mysql db
    $myConnection= mysqli_connect("localhost","root","******", "ii") or die ("could not connect to mysql"); 

    //read the json file contents
    $jsondata = file_get_contents('test.json');

    //convert json object to php associative array
    $data = json_decode($jsondata, true);

    $id = $data['device']['sn'];
    $ts = $data['data']['$ts'];
    $RH = $data['data']['RH'];
    $AT = $data['data']['AT'];
    $MINVi = $data['data']['MINVi'];
    $PTi = $data['data']['PTi'];
    $SDB = $data['data']['SDB'];
    $LWS = $data['data']['LWS'];
    $WSAV = $data['data']['WSAV'];
    $WSMX = $data['data']['WSMX'];
    $WSMN = $data['data']['WSMN'];
    $PR_TOT = $data['data']['PR_TOT'];
    $RAIN = $data['data']['RAIN'];
    $FDI = $data['data']['FDI'];
    $DT = $data['data']['DT'];
    $LAT = $data['data']['LAT'];
    $LON = $data['data']['LON'];
    $WD = $data['data']['WD'];
    $P1 = $data['data']['P1'];
    $AVGCi = $data['data']['AVGCi'];



    //insert into mysql table
    $sql = "INSERT INTO test(sn, date, RH, AT, MINVi, PTi, SDB, LWS, WSAV, WSMX, WSMN, PR_TOT, RAIN, FDI, DT, LAT, LON, WD, P1, AVGCi)
    VALUES('$id', '$ts', '$RH','$AT', '$MINVi', '$PTi', '$SDB', '$LWS', '$WSAV', '$WSMX', '$WSMN', '$PR_TOT', '$RAIN', '$FDI', '$DT', '$LAT', '$LON', '$WD', '$P1', '$AVGCi')";


 $query=mysqli_query($myConnection, $sql) or die(mysqli_error($myConnection)); 
?>

Tables Test data table and error_log table

JSON array var_dump - JSON var dump

Any help would be great

(After i get the general gist i want to incorporate PDO)

Answer 1

1. Do not convert json to associative array blindly. It creates more problems.
2. For accessing properties containing special characters or reserved words use placeholders like $data->{'$ts'}
3. Loop through arrays and objects if needed.
4. Adding an auto increment id column to tables helps to store data for one device.
5. It is a good idea to add time to error_log table as well

Tested bellow short version of your original question and it works.

<?php
        $_user = 'root';
        $_password= 'root';
        $_db = 'localtest';
        $_host = 'localhost';
        $_port = 3306;
    $con = new mysqli($_host, $_user, $_password, $_db) or die(mysql_error);

    //read the json file contents
    $jsondata = file_get_contents('test.json');

    //do not convert to array
    $json = json_decode($jsondata);

    $id = $json->device->sn;
    foreach($json->data as $key => $data){
        if(empty($data) || !isset($data->{'$ts'})){
            continue;
        }
        if (isset($data->{'$msg'})){
            $msg = $data->{'$msg'};
            $time = $data->{'$ts'};

            $sql="INSERT into error_log (sn, time, MSG) VALUES (?,?,?); ";
            $stmt = $con-> prepare($sql);
            $stmt -> bind_param("iss", $id,$time, $msg);
            $stmt -> execute();
        }else{
            $time = (isset($data->{'$ts'}))? $data->{'$ts'}:'';
            $RH = (isset($data->RH))? $data->RH:'';
            $AT = (isset($data->AT))? $data->AT:'';
            $MINVi = (isset($data->MINVi))? $data->MINVi:'';

            //insert into mysql table
            $sql="INSERT into test (sn, date, RH, AT, MINVi) VALUES (?,?,?,?,?); ";
            $stmt = $con-> prepare($sql);
            $stmt -> bind_param("issss", $id,$time,$RH,$AT,$MINVi);
            $stmt -> execute();
        }


    }
    mysqli_close($con);

?>

Answer 2

looks like the json decode is turning the array into one that is object based...run the $data through the function below to turn it into more of a relational array that is setup the way you are trying to read it....

I cannot find the function i had for this at the moment but If you read the array more like this:

$newvar = $data->node;

//to echo this try this...
echo '<pre style="text-align:left;">';
print_r($data);
echo '</pre>';

OR try fb logs for php to see the arrays in console of ya browser - use firephp for the addon.

it should work just fine...

Hope that helps.

Answer 3

Maybe something like this can get you on track... One thing I did notice is there doesn't appear to be an SDB entry in the given JSON data (FYI).

<?php
//connect to mysql db
$myConnection= mysqli_connect("localhost","root","******", "ii") or die ("could not connect to mysql"); 

//read the json file contents
$jsondata = file_get_contents('test.json');

//convert json object to php associative array
$data = json_decode($jsondata, true);

// Make sure $data has values
if (empty($data)) {
  // Process error here
}
else {
  // Make sure the proper keys have been set
  if (!(isset($data['device']) && !empty($data['device']) && isset($data['data']) && !empty($data['data']))) {
    // Process error here if not set
  }
  // If so, make sure the sn key has been set
  elseif (!isset($data['device']['sn']) || empty($data['device']['sn'])) {
    // Process error here if not set
  }
  else {        
    $data_arr = $data['data'];

    foreach ($data_arr as $key => $arr) {
      // Iterate through for loop
      for ($i = 0; $i < count($data_arr); $i++) {
        // Make sure every single key is set
        if (!(isset($data_arr[$i]['$ts']) && isset($data_arr[$i]['RH']) && isset($data_arr[$i]['AT']) && isset($data_arr[$i]['MINVi']) && isset($data_arr[$i]['PTi']) && isset($data_arr[$i]['SDB']) && isset($data_arr[$i]['LWS']) && isset($data_arr[$i]['WSAV']) && isset($data_arr[$i]['WSMX']) && isset($data_arr[$i]['WSMN']) && isset($data_arr[$i]['PR_TOT']) && isset($data_arr[$i]['RAIN']) && isset($data_arr[$i]['FDI']) && isset($data_arr[$i]['DT']) && isset($data_arr[$i]['LAT']) && isset($data_arr[$i]['LON']) && isset($data_arr[$i]['WD']) && isset($data_arr[$i]['P1']) && isset($data_arr[$i]['AVGCi']) && isset($data_arr[$i]['DT']) && isset($data_arr[$i]['DT']))) {
          // Process error here if not set
        }
        else {
          // If all is well, perform the query
          $id[$key][$i] = $arr['device']['sn'];
          $ts[$key][$i] = $data_arr[$i]['$ts'];
          $RH[$key][$i] = $data_arr[$i]['RH'];
          $AT[$key][$i] = $data_arr[$i]['AT'];
          $MINVi[$key][$i] = $data_arr[$i]['MINVi'];
          $PTi[$key][$i] = $data_arr[$i]['PTi'];
          $SDB[$key][$i] = $data_arr[$i]['SDB'];
          $LWS[$key][$i] = $data_arr[$i]['LWS'];
          $WSAV[$key][$i] = $data_arr[$i]['WSAV'];
          $WSMX[$key][$i] = $data_arr[$i]['WSMX'];
          $WSMN[$key][$i] = $data_arr[$i]['WSMN'];
          $PR_TOT[$key][$i] = $data_arr[$i]['PR_TOT'];
          $RAIN[$key][$i] = $data_arr[$i]['RAIN'];
          $FDI[$key][$i] = $data_arr[$i]['FDI'];
          $DT[$key][$i] = $data_arr[$i]['DT'];
          $LAT[$key][$i] = $data_arr[$i]['LAT'];
          $LON[$key][$i] = $data_arr[$i]['LON'];
          $WD[$key][$i] = $data_arr[$i]['WD'];
          $P1[$key][$i] = $data_arr[$i]['P1'];
          $AVGCi[$key][$i] = $data_arr[$i]['AVGCi'];

          $sql[$key][$i] = "INSERT INTO test(sn, date, RH, AT, MINVi, PTi, SDB, LWS, WSAV, WSMX, WSMN, PR_TOT, RAIN, FDI, DT, LAT, LON, WD, P1, AVGCi)
          VALUES('{$id[$key][$i]}', '{$ts[$key][$i]}', '{$RH[$key][$i]}','{$AT[$key][$i]}', '{$MINVi[$key][$i]}', '{$PTi[$key][$i]}', '{$SDB[$key][$i]}', '{$LWS[$key][$i]}', '{$WSAV[$key][$i]}', '{$WSMX[$key][$i]}', '{$WSMN[$key][$i]}', '{$PR_TOT[$key][$i]}', '{$RAIN[$key][$i]}', '{$FDI[$key][$i]}', '{$DT[$key][$i]}', '{$LAT[$key][$i]}', '{$LON[$key][$i]}', '{$WD[$key][$i]}', '{$P1[$key][$i]}', '{$AVGCi[$key][$i]}')";

          //insert into mysql table
          $query[$key][$i] = mysqli_query($myConnection, $sql[$key][$i]) or die(mysqli_error($myConnection));
        }
      }
    }
  }
}
?>

Answer 4

Consider building dynamic SQL calls conditionally depending on the error output. The array keys create the columns in INSERT clause and array values are quote wrapped in VALUES clause. Below echoes the sql statements just to demonstrate.

This should work for any change to JSON suffice no other new keys are added. Also, the array_splice is used to remove the $ts value the second time since you pass it in date column.

$jsondata = file_get_contents('Input.json');

$data = json_decode($jsondata, true);

$id = $data['device']['sn'];

foreach ($data['data'] as $k=>$v){
    if (array_key_exists("\$msg",$v) & !empty($v)){
       $sql = "INSERT INTO error_log (sn, msg)" 
              ." VALUES('$id', '". $v["\$msg"] ."')";
       echo $sql."\n";
       // INSERT INTO error_log (sn, msg) VALUES('5165654', 'SD_FAIL;1')
       // INSERT INTO error_log (sn, msg) VALUES('5165654', 'WDT;WV01')
       // INSERT INTO error_log (sn, msg) VALUES('5165654', 'WDT;SDI12')
       // INSERT INTO error_log (sn, msg) VALUES('5165654', 'WDT;LWS')

       $query = mysqli_query(...);
    } 
    if (!array_key_exists("\$msg",$v) & !empty($v)) {
       $keysArray = array_keys($v);
       $keysArray = array_splice($keysArray, 1);
       $vVals = array_splice($v, 1);

       $sql = "INSERT INTO test(sn, date, ". implode(", ", $keysArray) .")\n" 
              ." VALUES('$id', '". $v['$ts'] ."',". implode("', '", $vVals) .")";
       echo $sql."\n";

       // INSERT INTO test(sn, date, RH, AT, MINVi, PTi, LWS, WSAV, WSMX, WSMN,
       // PR_TOT, RAIN, FDI, DT, WD, P1, AVGCi)
       // VALUES('5165654', '170801170000',67.15', '12.87', '3.81', '23.4', '0*T',
       //  '0', '0', '0', '156', '0', '0.239', '2.881', '0*T', '0*T', '175)

       $query=mysqli_query(...)
    }
}