Parse JSON string from MySQL and INSERT name/value pairs into MySQL

Question

1. I have a column named 'other' in 'table_one' (MySQL) that contains a JSON string.

2. The table has millions of records.

3. A. I want to iterate through 'table_one', read from 'other' and B. INSERT all the $values into their corresponding $key columns in Table 2 ('table_two'). foreach $row (?)

4. I've already created the columns in 'table_2'.

5. I'd rather not (I think) create a static file (/path/some.json) from 'table_one' as it seems inefficient compared to simply reading straight from the database.

6. Some values in the JSON will be null, and some key/value pairs missing. So being able to pick up the $key(s) and $value(s) "dynamically" -- and not erroring out if they are absent -- is ideal/critical.

Sample of JSON in 'other':

{"firstName": "Bob", "lastName": "Jones", "email": "bob.jones@yahoo.com", "address": "7206 maplehurst drive", "city": "PORT RICHEY", "state": "FL", "zip": "34668", "ipAddress": "208.54.85.233", "gender": "M", "employer_name": "bobs auto", "months_address": "2", "years_address": "12"}

You'd (I'd) think by now there would be standard functionality to address this. If there is, again, my apologies.

Currently, I'm using an ETL tool to perform a simple, but monsterous, trim/split/map process. Each JSON string has 45 pairs. And any slight tweak in table structure requires re-mapping the whole thing.

*Comfortable with PHP, MySQL, JavaScript, jQuery.

Answer 1

A bit late for a question asked 110 days ago, but...

Take a look at common_schema, a MySQL add-on library: http://code.google.com/p/common-schema/ It has a extract_json_value function that uses an XPath specifier. Caveats are that it won't parse JSON arrays (just returns the array values as a space-delimited string) and doesn't seem to like unquoted number values in arrays.

And yes, common_schema does other cool stuff.

More robust is this UDF that can parse (well-formed) JSON strings: https://github.com/kazuho/mysql_json

Compiled easily on Ubuntu 10.04.4 LTS with MySQL 5.5.29, and it's quite fast. Caveats are that it returns the string "object" when it resolves to something that is not a value, and the need to specify a series of property keys is more awkward than common_schemas.extract_json_value's XPath specifier.

UPDATE: Alas, common_schema is a) closed, and b) incompatible with MySQL 5.7. Percona may update it at some point but I'm not holding my breath.

UPDATE: MySQL 5.7.8+ has native JSON fields and functions . The JSON functions use paths, are robust and fairly fast. There is a somewhat elaborate indexing opportunity using generated (stored) fields (hoping for index support of virtual fields). Caveats (as of 5.7.9): MySQL may reorder your keys, and subsequent values of duplicated keys are discarded (this is different from a lot of other JSON handlers that discard the prior values of duplicated keys).

Answer 2

I am not aware of standard function, bt I have done in 2 ways based on my mood.

Reliable method: Write a PHP script to Read from DB, convert json data to array using json_decode, and finally write back to DB

Quick method(when I m lazy do to in proper way): Using locate and substring get the values after the key till next key, something like below

substring(task_time_breakup, locate('","key1":"', task_time_breakup)+7, locate('""}',task_time_breakup)-locate('","key1":"', task_time_breakup)-7 )

Answer 3

mysql_json - a MySQL UDF for parsing JSON Works on Ubuntu 12.04.5 on gcc version 4.6.3 with MySQL 5.5

It has instruction how to install and examples of how to use it: https://github.com/ChrisCinelli/mysql_json