Invalid Syntax on ast.literal_eval() from data streaming result

Question

I'm using tweepy for streaming and json.loads to get the data. I saved it as txt files.

def on_data(self, data):
    all_data = json.loads(data)
    save_file.write(str(all_data)+"\n")

Now I want to extract several property from the data, but the problem is when I'm using ast.literal_eval() for solving the quotes and comma error, I'm getting another error.

    Traceback (most recent call last):

  File "C:\Users\RandomScientist\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-27-ffea75fc7446>", line 3, in <module>
    data = ast.literal_eval(data)

  File "C:\Users\RandomScientist\Anaconda3\lib\ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')

  File "C:\Users\RandomScientist\Anaconda3\lib\ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)

  File "<unknown>", line 2
    {'created_at': 'Thu Apr 04 07:00:10 +0000 2019', 'id': 1113697753530392577, 'id_str': '1113697753530392577', 'text': 'Karena kita adalah suratan terbuka kasih-Nya untuk dunia  \n#iamthemessenjah (link)', 'source': '<a href="http://www.facebook.com/twitter" rel="nofollow">Facebook</a>', 'truncated': False, 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 234355404, 'id_str': '234355404', 'name': 'Messenjah Clothing', 'screen_name': 'MessenjahCloth', 'location': 'YOGYAKARTA', 'url': 'http://www.messenjahclothing.com', 'description': 'THE WORLD CHANGER pages : http://www.facebook.com/messenjahclothingdotcom pin: 578CD443 WA: +6285 727 386 267 IG: @the_messenjah IG product @messenjahstore', 'translator_type': 'none', 'protected': False, 'verified': False, 'followers_count': 3405, 'friends_count': 190, 'listed_count': 7, 'favourites_count': 204, 'statuses_count': 58765, 'created_at': 'Wed Jan 05 13:13:10 +0000 2011', 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'lang': 'en', 'contributors_enabled': False, 'is_translator': False, 'profile_background_color': '7E808A', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme3/bg.gif', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme3/bg.gif', 'profile_background_tile': False, 'profile_link_color': '0400DB', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': '252429', 'profile_text_color': '666666', 'profile_use_background_image': True, 'profile_image_url': 'http://pbs.twimg.com/profile_images/882803510932156417/KenYVq-i_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/882803510932156417/KenYVq-i_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/234355404/1523949182', 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'quote_count': 0, 'reply_count': 0, 'retweet_count': 0, 'favorite_count': 0, 'entities': {'hashtags': [{'text': 'iamthemessenjah', 'indices': [60, 76]}], 'urls': [{'url': 'https: (link)', 'expanded_url': 'https://www.facebook.com/messenjahclothingdotcom/posts/2575823355778752', 'display_url': 'facebook.com/messenjahcloth…', 'indices': [77, 100]}], 'user_mentions': [], 'symbols': []}, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'filter_level': 'low', 'lang': 'in', 'timestamp_ms': '1554361210602'}
    ^
SyntaxError: invalid syntax

Here's my code

with open('pre-process.txt','r') as file:
    data = file.read()
    data = ast.literal_eval(data)
    print(data)

And I've been reading several answer like Python request using ast.literal_eval error Invalid syntax? and Python ast.literal_eval on dictionary string not working (SyntaxError: invalid syntax) but didn't get the suitable solution. Any idea? Thanks in advance.

Answer 1

It appears that you have one value per line in the file, so you need to read it a line at a time and call ast.literal_eval() on the line, not try to evaluate the entire file at once.

with open('pre-process.txt','r') as file:
    for line in file:
        data = ast.literal_eval(line)
        print(data)