Should I migrate local JSON database to MariaDB?

Question

Fictional story:

I have a car website containing over 200,000+ car listings around the United States. I get my data from two sources CarSeats and CarsPro updated nightly. Both sources contain about 100,000 detailed listings each in JSON format. The file size of both feeds is about 8GB and I plan to incorporate more used car sources in the near future.

The current JSON data contains everything I need to display car info from car search to car purchase, however, the JSON DB is stored locally and I use PHP's file_get_contents() to grab the appropriate metadata for each listing. It takes about 8 to 12 seconds to return 200 cars which is not bad, but I know there is room for improvement.

My Question:

Will migrating my data from localized JSON files to MariaDB 10.1 be a best practice move? Is that the scalable alternative for the future? What should my stack look like to improve speed and improve search capabilities?

Note:

• Forge installs MariaDB on the instance you boot.
• The 8GB is chunked by "car make" into over 20 different files. No individual file is larger than 400MB.

Currently using

• Laravel 5.2
• Forge
• PHP 5.6.10
• Servers from AWS

Answer 1

Will migrating my data from localized JSON files to MariaDB 10.1 be a best practice move? Is that the scalable alternative for the future? What should my stack look like to improve speed and improve search capabilities?

Yes. The whole purpose of a database is to make the storage—and usage—of data like this easier in the long run.

Each time you load a JSON file in PHP, PHP has to parse the data and I highly doubt 200,000 listings that consist of 8GB of data will ever work well as a file loaded into PHP memory from the file system. PHP will most likely die (aka: throw up an error) just when you attempt to load the file to begin with. Sorting and manipulating that data in PHP in that low-level state is even less efficient.

Storing that JSON data in a database of some kind—MariaDB, MySQL, MongoDB, etc…—is the only practical and best practice way to handle something like this.

The main reason anyone would repeatedly load a local JSON file into PHP would be for small tests and development ideas. On a practical level, it's inefficient but when you are in an early stage of development and don't feel like dealing with creating a process to import a large JSON file like that into an actual database a small sample file of data can be useful from your developer's perspective to hash out basic concepts and idea.

But there is utterly no “best practice” that would ever state a file being read from a filesystem is a “best practice”; it's honestly a very bad idea.

Answer 2

You will need Apache Solr that will improves the search and dealing with textual data.

The good point there that you will be able to use file_get_contents to deal with its query and its query results is in JSON format by default.