parse json from txt file not working in python

Question

I am trying to extract data from txt file in python which is a json dump. But I am getting JSONDecode Error

This is how I am adding json response into the file

repo=requests.get(url,headers=headers,params=params).json()
if repo:
    with open('data.txt', 'a') as f:
        json.dump(repo, f, sort_keys=True, indent=4)
    continue
else:
    break

this is my json structure

[
    {
        "login": "asu",
        "login_name": "heylo"
    },
    {
        "login": "sr9",
        "login_name": "heylo"
    }
],
[
    {
        "login": "tokuda109",
        "login_name": "mojombo"
    },
    {
        "login": "svallory",
        "login_name": "mojombo"
    }
]

this is I am trying to extract

with open('data.txt') as fd:
    json_data = json.load(fd)
    pprint(json_data)

Answer 1

As mentioned in the comment, just concatenating JSON objects into a file one after another does not make a file that's valid JSON (and that could be parsed as a single JSON object).

The minimal better format is JSON Lines, https://jsonlines.org/ , that is a file of lines that are each a JSON document.

You can create such a file by appending to a file while ensuring indent is off when dumping JSON:

with open('data.txt', 'a') as f:
    print(json.dumps(repo, sort_keys=True), file=f)

Using print() ensures a trailing newline.

Then, you can load the data with eg

with open('data.txt') as fd:
    json_data = [json.loads(line) for line in fd if line.strip()]

If the current file of concatenated JSON documents is important to you, you could try and repair it with a hack like wrapping the file's contents with [ and ] and adding a comma between otherwise malformed concatenated documents, but this is not quite guaranteed to work.

with open('data.txt') as fd:
    fixed_json_data = json.loads("[" + fd.read().replace("}{", "},{") + "]")

Answer 2

As described in How to append data to a json file? , using a mode is not a good choice, I think it is better to append fetched data to the available list in data.txt manually like this:

import json
import requests


def read_content():  # reads and returns the available list in data.txt
    try:
        with open('data.txt') as fd:
            json_data = json.load(fd)
    except:
        json_data = []  # handle the first write, when the file does not exist
    return json_data


url = 'https://api.github.com/users/sferik/followers?per_page=100&page=1'
repo = requests.get(url).json()  # repo is a list

if repo:
    available_content = read_content()  # read available list in data.txt
    available_content.extend(repo)  # extend new list to the end of available list
    with open('data.txt', 'w') as f:  # write again, the mode is 'w'
        json.dump(repo, f, sort_keys=True, indent=4)