Write and read specific variables from external file in Python

Question

I'm writing a program where I would like to both read and write specific variables with different data types from/to an external file. The closest I've gotten after trying a few different modules, is by using pickle. Pickle seems great as it understands the different data types, but it falls short as I understand it reads line from line from the top instead of being able to call specific variables by name as you can from an external .py file.

This module, and other modules, also seem to overwrite the whole file if you write new or change existing variables, so you would have to rewrite all the data if you'd actually only like to change one of them.

See code example down below. Sorry for the long code, I just wanted to be thorough in my explanation.

In this particular program it doesn't matter if the file is humanly readable or not. Could anyone point me in the direction of a module that can handle this, or tell me what I might be doing wrong?

import pickle

variable1 = "variable1"
variable2 = "variable2"

pickle_out = open("db.pkl","wb")
pickle.dump(variable1, pickle_out)
pickle.dump(variable2, pickle_out)
pickle_out.close()

#So I'll load the variables in again

pickle_in = open("db.pkl", "rb")
variable1 = pickle.load(pickle_in)
variable2 = pickle.load(pickle_in)
print(variable2)
variable2

#Everything good so far.
#But let's say I only want to load variable2 because I can't remember which 
#line it was written on.

pickle_in = open("db.pkl", "rb")
variable2 = pickle.load(pickle_in)
print(variable2)
variable1

#Also, if I'd like to update the value of variable1, but leave the other 
#variables untouched, it wouldn't work as it would just overwrite the whole 
#file.

#Let's say I've loaded in the variables like at the line 17 print statement.

variable1 = "variable1_new"
pickle_out = open("db.pkl","wb")
pickle.dump(variable1, pickle_out)
pickle_out.close()
pickle_in = open("db.pkl", "rb")
variable1 = pickle.load(pickle_in)
variable2 = pickle.load(pickle_in)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
EOFError: Ran out of input

print (variable1)
variable1_new

#So the value of variable1 is correct, but variable2 is no longer in the 
#pickle-file as the whole file was overwritten.

Answer 1

Based on your comment that the data stored is relatively simple, a JSON file may be easier to work with.

Consider the following file config.json :

{
    "str_var": "Somevalue1",
    "list_var": ["value2", "value3"],
    "int_var": 1,
    "nested_var": {
        "int_var": 5
    }
}

Now you can read and use the values as follows

import json

# With statement will close the file for us immediately after line 6
with open("config.json", "r") as config_file:
    # Load the data from JSON as a Python dictionary
    config = json.load(config_file)

# See all top level values
for key, value in config.items():
    print(key, ":", value, "Type", type(value))

# Use individual values
print(config["str_var"])
print(config["list_var"][0])
print(config["nested_var"]["int_var"])

# Do something constructive here
...

# Update some values (e.g. settings or so)
config["str_var"] = "Somevalue2"
config["int_var"] = 5
config["list_var"].append("value4")

# Write the file
json.dump(config, open("config.json", "w"), indent=4, sort_keys=True)

Resulting in the following JSON file:

{
    "int_var": 5,
    "list_var": [
        "value2",
        "value3",
        "value4"
    ],
    "nested_var": {
        "int_var": 5
    },
    "str_var": "Somevalue2"
}

This allows you to eg load such a config file, and you are free to use any value in it specifically, no need to know the index. Hope this gives you an idea how to handle your data using JSON.