将BIG MySQL表导出为JSON

Question

I have a mysql table with 2.8 million records and i want to convert all of these to JSON. I wrote a script to convert but it stops with a memory warning.

Then i tried to create smaller files (file1 is 0, to 100000 records, file 2 is 100000 to 1000000 records etc ) and combine with windows copy command. It works, but each file is a JSON array (like [{...}]) and when it merges, it becomes separate sections like [{}][{}] (where i want it like [{................}])

Is there any better solution to do this ?

Answer 1

I would suggest you to change 'memory_limit' in your php.ini configuration. Also if this takes much time then you can handle this by cron job(if possible)

OR

you can decode your ALL json files and merge it in a single array and then again encode in json and put in the json file.

Answer 2

Finally i did this. Please see the steps (i am not sure this is the right one, but it works).

Totally, i have 2.6 million records in my table. I created a script which will select mysql rows, convert to json and write to a file.

Select records from 0 to 1 million and create file 1. Repeat from 1 to 2 million and 2 to 2.6 million for file2 and file 3.

Combine these files using JQ ( http://stedolan.github.io/jq/ ) and create a single JSON file.