Formatting and Decoding Twitter Stream JSON Output

Question

I've having trouble formatting and encoding the twitter stream that is being collected with a python script I wrote. The output looks like this:

{"created_at":"Wed May 07 20:53:05 +0000 2014", "id":464145921098674177, "id_str":"464145921098674177" ...

... and continues with this single entry along one line. Each line is a single tweet with massive amounts of information structured just the same.

I've tried simply using python's JSON module to turn the json file into a dict, but it keeps giving me an error - stating that the structure isn't a in JSON serialization.

Ultimately, I'd like to feed the JSON output into a table format. I'm trying to get the file into a csv and go from there. I'd settle for anything readable at this point. FYI - I'm trying to stick to Python because it's what I know.

Here's the python code I tried to use:

import json

json_file = open('twitterOutput.json', 'r').readlines()

j = json.loads(json_file[0])

print j

Which gives me the error: "No JSON object could be decoded". FYI - this is just test code. I just wanted to try to get one of the lines of the json_file list to work.

Thanks.

Answer 1

It's hard to tell where you are going wrong without seeing any code, but the following should do it:

import json

twitter_output = # string of twitter output
twitter_output_dict = json.loads(twitter_output)

Or, if the output is stored in a file, then:

import json

with open('twitter_output.json') as twitter_output_file:
    twitter_output_dict = json.load(twitter_output_file)

As for "trying to get the file into a csv and go from there", you would have to explain how you want to approach this and how you want it to be structured. As far as I know, the Twitter-returned JSON is a nested structure (as seen on Twitter's docs so displaying it in csv format really depends on how you want to structure it.