Append json files while iterating through for loop

Question

I want to iterate through some range of pages and save all of them into one json file, that is append page 2 to page 1 and page 3 to already appended page2 to page1.

for i in range(4):
    response = requests.post("https://API&page="+str(i))
    data = response.json()
    my_data = json.load(open( "data.json" ))
    my_data.update(my_data)
    json.dump(data, open( "data.json", 'w' ))

Basing on some answers from similar question I wrote something like that, but it overwrites instead of appending one page to another.

The json data structure is as follows:

ending with page number that increments every page.

Any idea what I did wrong?

Answer 1

Your code is overwriting the contents of the data.json file on each iteration of the loop. This is because you are using the 'w' mode when calling json.dump , which will overwrite the contents of the file.

To append the data to the file, you can use the 'a' mode instead of 'w' when calling json.dump . This will append the data to the end of the file, rather than overwriting the contents.

Like this

for i in range(4):
    response = requests.post("https://API&page="+str(i))
    data = response.json()
    my_data = json.load(open( "data.json" ))
    my_data.update(my_data)
    json.dump(data, open( "data.json", 'a' ))

Answer 2

What is it that you are trying to achieve?

You are overwriting the file data.json each time with the result of the response saved in the variable data .

Your code has 2 issues: you are updating a dictionary and you are overwriting a file. Any of the two could solve your problem, depending on what you want to achieve.

It looks like you instead want to save the contents of my_data like that:

json.dump(my_data, open( "data.json", 'w' ))

Anyway, my_data will be a dictionary that gets its contents overwritten each time. Depending on the structure of data , this could not be what you want.

I'll explain better: if your structure is, for any page, something like

{
    "username": "retne",
    "page": <page-number>
}

my_data will just be equal to the last data page.

Moreover, about the second issue, if you open the file in 'w' mode, you will always overwrite it. If you will open it in 'a' mode, you will append data to it, obtaining something like this:

{
    "username": "retne",
    "page": 1
}
{
    "username": "pentracchiano",
    "page": 2
}
{
    "username": "foo",
    "page": 3
}

but this is not a valid .json file, because it contains multiple objects with no delimiters.

Try being clearer about your intents and I can provide additional support.