将JSON文件写入一个txt文件中，每个JSON数据都在自己的行中

Question

So I am querying Twitter API with a list of tweet IDs. What I need to do is looping through the IDs in order to get the corresponding data from Twitter. Then I need to store those JSON files into a txt file where each tweet's JSON data is on its own line. Later I will have to read the txt file line by line to create a pandas df from it.

I try to give you some fake data to show you the structure.

twt.tweet_id.head()

0    000000000000000001
1    000000000000000002
2    000000000000000003
3    000000000000000004
4    000000000000000005
Name: tweet_id, dtype: int64

I don't know how to share the JSON files and I don't even know if I can. After calling tweet._json what I get is a JSON file.

drop_lst = []     # this is needed to collect the IDs which don't work


for i in twt.tweet_id:   # twt.tweet_id is the pd.series with the IDs
    try:
        tweet = api.get_status(i)
        with open('tweet_json.txt', 'a') as f:
            f.write(str(tweet._json)+'\n')  #  tweet._json is the JSON file I need

    except tp.TweepError:
        drop_lst.append(i)

the above works but I think I have lost the JSON structure which I need later to create the dataframe

drop_lst = []

for i in twt.tweet_id:
    try:
        tweet = api.get_status(i)
        with open('data.txt', 'a') as outfile:  
            json.dump(tweet._json, outfile)

    except tp.TweepError:
        drop_lst.append(i)

the above doesn't put each file on its own line.

I hope I was able to provide you with enough information to help me.

Thank you in advance for all your help.

Answer 1

Appending json to a file using json.dump doesn't include newlines, so they all wind up on the same line together. I'd recommend collecting all of your json records into a list , then use join and dump that to a file

tweets, drop_lst = [], []

for i in twt.tweet_id:
    try:
        tweet = api.get_status(i)
        tweets.append(tweet._json)

    except tp.TweepError:
        drop_lst.append(i)

with open('data.txt', 'a') as fh:
    fh.write('\n') # to ensure that the json is on its own line to start
    fh.write('\n'.join(json.dumps(tweet) for tweet in tweets)) # this will concatenate the tweets into a newline delimited string

Then, to create your dataframe, you can read that file and stitch everything back together

with open("data.txt") as fh:
    tweets = [json.loads(line) for line in fh if line]

df = pd.DataFrame(tweets)

This assumes that the json itself doesn't have newlines, which tweets might contain, so be wary