Optimizing the execution time of the foreach loop

Question

I use XMLStreamer for reading the XML file ( Structure here ).

There are many products in 1 XML file (about 100,000 per file). The products have parameters, in total there are up to 500,000 of them in the file. There will be even around 20 XML files (maybe more, each with different products). Will the database be a good place to store such data?

Main problem:

The problem is the execution time of the script that enters the data into the database. Now with around 35,000 products, the time is around 300s. How to optimize it?

Code:

if ($result = $mysqli->query('SELECT id FROM products WHERE shop_id = 1')) {
    if ($result->num_rows > 0) {
        $result->free_result();
        echo "Find results...\n"; // for test
    } else {
        echo "No results, so we can insert new records...\n"; // for test
        $streamer = new XMLStreamer('offer', 'products', 'product');
        $mysqli->begin_transaction();
        $start = time();
        try {
            $mysqli->query('SET FOREIGN_KEY_CHECKS=0;');
            foreach ($streamer->stream('bds.xml') as $product) {
                $document = new \DOMDocument();
                $document->appendChild($product);
                $element = simplexml_import_dom($product);
                $productId = $element->attributes()->id;
                $productName = $element->description->name;
                $sql = '
                    INSERT INTO products (shop_id, product_id, product_name)
                        VALUES(
                            ' . $shopId . ',
                            ' . $productId . ',
                            "' . $mysqli->real_escape_string($productName) . '")
                        ON DUPLICATE KEY UPDATE
                            shop_id = ' . $shopId . ',
                            product_id = ' . $productId . ',
                            product_name = "' . $mysqli->real_escape_string($productName) . '";
                ';
                $mysqli->query($sql);
                foreach ($element->parameters as $parameters) {
                    foreach ($parameters as $parameter) {
                        $parameterId = (int) $parameter->attributes()['id']->__toString();
                        $parameterName = $mysqli->real_escape_string($parameter->attributes()->{'name'}->__toString());
                        $sql = 'INSERT INTO parameters (shop_id, product_id, parameter_id, parameter_name)
                                VALUES(
                                    ' . $shopId . ',
                                    ' . $productId . ',
                                    ' . $parameterId . ',
                                    "' . $parameterName . '")
                                ON DUPLICATE KEY UPDATE
                                    shop_id = ' . $shopId . ',
                                    product_id = ' . $productId . ',
                                    parameter_id = ' . $parameterId . ',
                                    parameter_name = "' . $parameterName . '";
                        ';
                        $mysqli->query($sql);
                        if (isset($parameter->{'value'})) {
                            foreach ($parameter->{'value'} as $parameterValue) {
                                $valueId = (int) $parameterValue->attributes()->{'id'}->__toString();
                                $valueName = $mysqli->real_escape_string($parameterValue->attributes()->{'name'}->__toString());
                                $sql = 'INSERT INTO parameter_values (shop_id, parameter_id, value_id, value_name)
                                    VALUES(
                                        ' . $shopId . ',
                                        ' . $parameterId . ',
                                        ' . $valueId . ',
                                        "' . $valueName . '")
                                    ON DUPLICATE KEY UPDATE
                                        shop_id = ' . $shopId . ',
                                        parameter_id = ' . $parameterId . ',
                                        value_id = ' . $valueId . ',
                                        value_name = "' . $valueName . '";
                            ';
                                $mysqli->query($sql);
                            }
                        }
                    }
                }
                $mysqli->query('SET FOREIGN_KEY_CHECKS=1;');
                $productIndex++;
            }
            $mysqli->commit();
            var_dump("Time...:  " . (time() - $start));
        } catch (mysqli_sql_exception $exception) {
            $mysqli->rollback();
            throw $exception;
        }
    }
}

Database structure:

Answer 1

Batch insert.

Instead of inserting one row at a time, insert 100 in a single INSERT . This may shrink the 300 seconds to 30.

Re-think the PRIMARY KEY of the tables. In order to do IODKU, you need to not specify the set of unique columns in the UPDATE part while specifying them in the INSERT part. I would guess that some combination of the columns is really the PK.