搜索数百万个JSON文件的最佳方法是什么？

Question

I've very recently picked up programming in Python and am working on creating a database.

I've already worked out extracting all these files from their source so they are all in a directory on my computer.

All of these files are structured the same way and what I want to do is search these multidimensional dictionaries and locate the value for a specific set of keys.

These json files are all structured similarly,

{
"userid": 34535367,
"result": {
    "list": [
        {
            "name": 264,
            "age": 64,
            "id": 456345345
        },
        {
            "name": 263,
            "age": 42,
            "id": 364563463456
        }
    ]
}

}

In my case, I would like to search for the "name" key and return the relevant data(quality, id and the original userid) for the thousands of names just like it from my millions of JSON files.

Basically I'm very new at this and the little programming knowledge I have is in Python. I'm happy to start learning whatever I need to, but I'm not sure which direction to go.

Answer 1

If your goal is to create a database, then you should look on how databases work and solve the same problem you are trying to solve right now :)

NoSQL databases (like mangodb) work also with json documents and implements most likely a whole set of tools to search and filter documents.

Now to answer your question, there is no quick way to do so unless you do some preprocessing, meaning that you store different information about the data (called metadata). This is a huge subject and I don't have enough expertise to give you all the answers, but I can give you a simple tip: Use indexes.

An index is a sorted key/value map where for every value, we store the documents that contains that value (or the file + position of the Json document) . For example an index for the name property would like this:

{
263: ('jsonfile10.json', '0')
264: ('jsonfile10.json', '30'), 
# The json document can be found on the jsonfile10.json file on line 30
}

By keeping an index for the most queried values, you can turn a linear time search into a logarithmic time search not to mention that inserting a new document is much faster. in your case, you seems to only need an index on the name field.

Creating/updating the index is done when you insert, update or remove a document. Using a balanced binary tree can accelerate the updates on the index.

Answer 2

As a suggestion, why don't you just process all the incoming files and insert the data into a database? You will have a toolset to query that database. SQLite for example will do (as well as any other more sophisticated database): http://www.sqlite.org/ http://docs.python.org/2/library/sqlite3.html

Simple other solution might be to build a file mapping name_id to /file/path . Then you can logarithmically do a binary search by the name id. But I'd still advise using a proper database as maintaining the index will be more cumbersome than doing some inserts/deletes.