MySql：追加到数据库中当前值的最快方法

Question

I have a table with 32,000 entries.

Each entry has a JSON blob against it:

{
    "property1": "someValue",
    "property2": "AnotherValue"
}

I now need to append another property to this blob for every entrant so that we have:

{
    "property1": "someValue",
    "property2": "AnotherValue",
    "property3": "YetAnotherValue"
}

Because someValue and AnotherValue will be different for different entries I cannot just replace the old records with the new one. Instead (I'm guessing) I'm supposed to parse JSON, add property3 and return it to DB for each of the 32,000 records.

What is the best way to achieve this? I have a MySql database which I interact with through PhpMyAdmin.

I tried writing a PHP script which will basically select and loop through every row, parse and append to my JSON blob and put it back up for every record. However after I run this in a browser it simply times out on me because it takes too long to execute (I suspect it's the wait time between pulling the blob and uploading it back up for every record).

Any help is greatly appreciated!

Answer 1

I would question the need to save JSON into a MySQL DB directly. Let's say your table with the JSON blob is called my_table and it has an id field. I'd create another table, say, my_json_properties . There, I'd create three columns: my_table_id , property_name and property_value . Then just populate that table with the JSON properties, and adding new properties is trivial. Also remember to delete the blob column and of course to update your data access objects.

Note: use the InnoDB storage engine so you get to make the my_json_properties.my_table_id into a foreign key that references my_table.id . MySQL also automatically creates an index with the foreign key columns, so it's very effective to query the properties when you just know the id in the original table.

Answer 2

You could use an UPDATE statement to add this new property to your json blob, if the json is regular, ie there's no blank after the last property and the } has got a fixed position in the next line. Building on your example and assuming \\n as end of line you could do it like that:

UPDATE example
SET json = REPLACE(
               json, 
               '"\n}', 
               '",\n    "property3": "some value"\n}');

Demo( http://sqlfiddle.com/#!2/db972/1 )

A more robust approach would be by the suggestions of Marcus Adams:

UPDATE example
SET json = 
    CONCAT(
        TRIM(
            TRAILING FROM TRIM(
                TRAILING '\n' FROM TRIM(
                    TRAILING FROM SUBSTRING_INDEX(json, '}', 1)
                )
            )
        ),
        ',\n    "property3": "some value"\n}'
    );

This will work independently of trailing spaces after the last property and leading spaces before the } , but it's still relying on the \\n as end of line.

Note:

Such statements are usually much faster than row by row updating with a script.

Answer 3

It depends on use case. If this is a one time task, it would be MUCH better to run your script from the shell and not from browser to avoid web server and browser timeouts.

php -f my_update_script.php

If you cannot run script from the shell for whatever reason you can try to fool the browser and web server as follows:

set_time_limit(3600*2); // two hours should be enougth?
// turn off output buffering
while (ob_end_flush());
ob_implicit_flush();

// Your mysql fetch..update loop here
// begin loop
echo 'Something useful every iteration to let them know your script is alive :)';
// end loop

However this approach may fail if there is caching reverse proxy in front of your web server.

MariaDb has native support for JSON format (since version 10), you may consider to switch to MariaDb.