在PHP中的JSON文件中是否有更高效（更快）的搜索和匹配值的方法？

Question

I have an API that allows the search of information based on an order number like so:

file_get_contents('https://www.example.com/api/status.php?os='.$orderId)

status.php has the following code:

if (isset($_GET['os'])) { $orderId = strip_tags(htmlspecialchars($_GET['os'])); }

if (isset($orderId)) {

    try {

        $array = \JsonMachine\JsonMachine::fromFile($file);
        // using JSON Machine https://github.com/halaxa/json-machine

        $result = getOrderStatus($orderId, $array);

        if (empty($result)) {

            http_response_code(206);

        } else {

            echo json_encode($result, JSON_UNESCAPED_UNICODE);

        }

    } catch (Exception $e) {

        echo json_encode(array(
            'error' => array(
                'code' => $e->getCode(),
                'message' => $e->getMessage()
            )
        ));

    }

} 

The above loads the json file (currently 1.4MB, approx 3000 objects, 15 key/value pairs per object) and passes the whole decoded json to the getOrderStatus function:

function getOrderStatus($orderId, $array) {

    $resultArray = array();

    foreach ($array as $val) {

        $oid = explode(' ', $val['ORDER TITLE']);
        $oid = $oid[0];
        $oid = explode('_', $oid);
        $oid = $oid[0];

        if ($oid == $orderId) {

            $resultArray['status'] = $val['STAT ID'];

            $resultArray['email'] = $val['STAT CTRL'];

            $resultArray['mzId'] = $val['ORDER ID'];

            $resultArray['mcId'] = $val['COMPANY ID'];

            $resultArray['count'] = $val['NUM PCS'];

        }

    }

    return $resultArray;

}

if ($oid == $orderId) matches whether the id from the API matches the value of the key "ORDER TITLE", in which case the rest of the information in that object is relevant to the search.

Here is an example of two objects from the JSON file:

[
   {
      "INTERNAL ID": "914693",
      "ORDER TITLE": "0108491 A_PRODUCT_NAME",
      "COMPANY ID": "",
      "STAT ID": "1.2",
      "STAT CTRL": "example@example.com",
      "POST ID": "Post",
      "SML": "Transfer",
      "UPDATE": "17.06.2019 10:52:45",
      "TOTAL": "0",
      "NUM PCS": "1",
      "PAID": "",
      "TEXT": "",
      "PROBLEM": ""
   },
   {
      "INTERNAL ID": "914694",
      "ORDER TITLE": "0108494 A_PRODUCT_NAME",
      "COMPANY ID": "",
      "STAT ID": "1.2",
      "STAT CTRL": "example@example.com",
      "POST ID": "Post",
      "SML": "Transfer",
      "UPDATE": "17.06.2019 10:52:45",
      "TOTAL": "0",
      "NUM PCS": "1",
      "PAID": "",
      "TEXT": "",
      "PROBLEM": ""
   }
]

The problem is that I have no control over the format of the JSON file, since it is an export from external software.

Currently, I am searching and matching approx. 100 order numbers. This means that with the above code, for each order number - the API is called, the JSON file needs to be opened, each of 3000 objects tested for the match, array with info returned, all 100 times. This is why I think the process takes 40 seconds. While this is bearable, there could be as much as 10x more objects in the JSON file in the future.

What can do to make the search faster? I was thinking of calling the API with an array of order numbers, then opening the file once and matching each order number. Would this be the correct approach?

Answer 1

What I ended up doing may be overly complicated, but serves the purpose well and should be resilient to the main json file getting larger in the future. For some background: The web app is used for an overview of orders and detailed information about the orders, which means touching a few local and external APIs. Approx. 5 people (5 browsers) may be using the app at any given time.

External software uploads a large json file with most of the information via ftp.

To automate the entire process, I found out about incron , which is similar to cron , but instead of acting on time, it acts on file/directory manipulation (creation, update, deletion, etc.). This of course requires ssh and root access to the local server.

sudo apt install incron

incrontab -e

/route/to/json/file/big.json IN_CLOSE_WRITE /usr/bin/php /route/to/php/load.php

load.php runs every time big.json is changed.

To shorten the load time, I created a database table with columns based on big.json . load.php reads big.json and saves each object as a row in the database table. The part of the code that provides this info via API now reads that info from the database instead of big.json , as suggested in the comments. This brought down the time to 22s, instead of 44s.

Although just converting json to mysql helped, I thought it was unnecessary for the individual browsers to keep repeating the load process, so instead incron does pretty much the same thing, but once and only upon file change.

load.php can filter unneeded data and also loads all external APIs and combines the data in correlation to the data from big.json , then saves everything to a new file called data-full.json . The size of this resulting file is about 16x smaller than big.json . The browser app with datatables.js then loads this file in a matter of milliseconds.

So in short:

• Load large json file only upon its update with incron
• Convert large json file to mysql
• Create a new json file combining filtered mysql data and external API data
• Ajax call the "static" new json file in datatables.js
• Add timestamp of when the new json file was last updated in php with filemtime()