Python: How should I save many dictionaries to disk?

Question

I'm indexing thousands of jsons and want to save them to disk and be able to access them quickly later. What types of solutions should be looking at? Would there be any advantage to keeping them all as separate jsons? Is there an easy way to get this data neatly into sql? Should I be looking at a non-relational db?

The jsons are nested dictionaries, some with large tables in them. Others parts of the json have one entry and can easily be stored as a table. Is there some easy way to un-nest all of the tables and keep those separate from the single row's of metadata within the file?

Answer 1

if each json is indexably by a key and if you mean with json a string with json syntax, then you might use the dbm module https://docs.python.org/3.5/library/dbm.html#module-dbm

If you mean with json a structure, that was loaded from json or a structure, that can be converted to a json string, then you might use shelve https://docs.python.org/3.5/library/shelve.html

example with dbm:

with dbm.open('myjsons', 'c') as db:
    db["key1"] = '{ "a": "value", "b", "othervalue"}'
    db["key2"] = '[ 1, 2, "three", 4.0 ]'

    jsonstr = db.get("key1", None)

example with shelve

with shelve.open('spam') as db:
        db["key1"] = { "a": "value", "b", "othervalue"}
        db["key2"] = [ 1, 2, "three", 4.0 ]

        jsonstr = db.get("key1", None)

Please note, that you cannot modify the data from two processes at a time.

If that would be required you might have to look at sqlitedict, mongodb, couchdb