Convert string lines to valid json format in Python

Question

Given a test.json file with content as follows:

{'review/appearance': 2.5, 'beer/style': 'Hefeweizen', 'review/palate': 1.5, 'review/taste': 1.5, 'beer/name': 'Sausa Weizen', 'review/timeUnix': 1234817823, 'beer/ABV': 5.0, 'beer/beerId': '47986', 'beer/brewerId': '10325', 'review/timeStruct': {'isdst': 0, 'mday': 16, 'hour': 20, 'min': 57, 'sec': 3, 'mon': 2, 'year': 2009, 'yday': 47, 'wday': 0}, 'review/overall': 1.5, 'review/text': 'A lot of foam. But a lot.\tIn the smell some banana, and then lactic and tart. Not a good start.\tQuite dark orange in color, with a lively carbonation (now visible, under the foam).\tAgain tending to lactic sourness.\tSame for the taste. With some yeast and banana.', 'user/profileName': 'stcules', 'review/aroma': 2.0}
{'review/appearance': 3.0, 'beer/style': 'English Strong Ale', 'review/palate': 3.0, 'review/taste': 3.0, 'beer/name': 'Red Moon', 'review/timeUnix': 1235915097, 'beer/ABV': 6.2, 'beer/beerId': '48213', 'beer/brewerId': '10325', 'review/timeStruct': {'isdst': 0, 'mday': 1, 'hour': 13, 'min': 44, 'sec': 57, 'mon': 3, 'year': 2009, 'yday': 60, 'wday': 6}, 'review/overall': 3.0, 'review/text': 'Dark red color, light beige foam, average.\tIn the smell malt and caramel, not really light.\tAgain malt and caramel in the taste, not bad in the end.\tMaybe a note of honey in teh back, and a light fruitiness.\tAverage body.\tIn the aftertaste a light bitterness, with the malt and red fruit.\tNothing exceptional, but not bad, drinkable beer.', 'user/profileName': 'stcules', 'review/aroma': 2.5}
{'review/appearance': 3.0, 'beer/style': 'Foreign / Export Stout', 'review/palate': 3.0, 'review/taste': 3.0, 'beer/name': 'Black Horse Black Beer', 'review/timeUnix': 1235916604, 'beer/ABV': 6.5, 'beer/beerId': '48215', 'beer/brewerId': '10325', 'review/timeStruct': {'isdst': 0, 'mday': 1, 'hour': 14, 'min': 10, 'sec': 4, 'mon': 3, 'year': 2009, 'yday': 60, 'wday': 6}, 'review/overall': 3.0, 'review/text': 'Almost totally black. Beige foam, quite compact, not bad.\tLight smell, just a bit of roast, and some hop. A bit too light.\tThe taste is light oo, and drinkable, with some malt, roast, hints of coffee.\tNothing exceptional, but after all drinkable and pleasant.\tLight to average body.\tIn the aftertaste some dust, somr roast, hint of caramel, and a bit of bitterness.\tNo defect, drinkable, not bad.', 'user/profileName': 'stcules', 'review/aroma': 2.5}
{'review/appearance': 3.5, 'beer/style': 'German Pilsener', 'review/palate': 2.5, 'review/taste': 3.0, 'beer/name': 'Sausa Pils', 'review/timeUnix': 1234725145, 'beer/ABV': 5.0, 'beer/beerId': '47969', 'beer/brewerId': '10325', 'review/timeStruct': {'isdst': 0, 'mday': 15, 'hour': 19, 'min': 12, 'sec': 25, 'mon': 2, 'year': 2009, 'yday': 46, 'wday': 6}, 'review/overall': 3.0, 'review/text': 'Golden yellow color. White, compact foam, quite creamy. Good appearance.\tFresh smell, with good hop. Quite dry, with a good grassy note. Hay. Fresh and pleasant.\tMore sweet in the mouth, with honey. The hop comes back in the end, and in the aftertaste.\tNot bad, but a bit too sweet for a pils.\tIn the end some vanilla and camomile note.\tIn the aftertaste, too. Though the hop, a bit too sweet.\tHonest.', 'user/profileName': 'stcules', 'review/aroma': 3.0}
{'review/appearance': 4.0, 'beer/style': 'American Double / Imperial IPA', 'review/palate': 4.0, 'review/taste': 4.5, 'beer/name': 'Cauldron DIPA', 'review/timeUnix': 1293735206, 'user/gender': 'Male', 'user/birthdayRaw': 'Jun 16, 1901', 'beer/ABV': 7.7, 'beer/beerId': '64883', 'user/birthdayUnix': -2163081600, 'beer/brewerId': '1075', 'review/timeStruct': {'isdst': 0, 'mday': 30, 'hour': 18, 'min': 53, 'sec': 26, 'mon': 12, 'year': 2010, 'yday': 364, 'wday': 3}, 'user/ageInSeconds': 3581417047, 'review/overall': 4.0, 'review/text': "According to the website, the style for the Caldera Cauldron changes every year. The current release is a DIPA, which frankly is the only cauldron I'm familiar with (it was an IPA/DIPA the last time I ordered a cauldron at the horsebrass several years back). In any event... at the Horse Brass yesterday.\t\tThe beer pours an orange copper color with good head retention and lacing. The nose is all hoppy IPA goodness, showcasing a huge aroma of dry citrus, pine and sandlewood. The flavor profile replicates the nose pretty closely in this West Coast all the way DIPA. This DIPA is not for the faint of heart and is a bit much even for a hophead like myslf. The finish is quite dry and hoppy, and there's barely enough sweet malt to balance and hold up the avalanche of hoppy bitterness in this beer. Mouthfeel is actually fairly light, with a long, persistentely bitter finish. Drinkability is good, with the alcohol barely noticeable in this well crafted beer. Still, this beer is so hugely hoppy/bitter, it's really hard for me to imagine ordering more than a single glass. Regardless, this is a very impressive beer from the folks at Caldera.", 'user/profileName': 'johnmichaelsen', 'review/aroma': 4.5}

How could I convert it to standard json format? Thanks.

My trial:

import json

with open('./test.json','r') as f:
    s = f.read()
    s = s.replace('\t','')
    s = s.replace('\n','')
    s = s.replace("\'", "\"")
    data = json.load(s)
    print(s)

Output:

AttributeError: 'str' object has no attribute 'read'

Expected result:

[{"review/appearance": 2.5, "beer/style": "Hefeweizen", "review/palate": 1.5, "review/taste": 1.5, "beer/name": "Sausa Weizen", "review/timeUnix": 1234817823, "beer/ABV": 5.0, "beer/beerId": "47986", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 16, "hour": 20, "min": 57, "sec": 3, "mon": 2, "year": 2009, "yday": 47, "wday": 0}, "review/overall": 1.5, "review/text": "A lot of foam. But a lot.\tIn the smell some banana, and then lactic and tart. Not a good start.\tQuite dark orange in color, with a lively carbonation (now visible, under the foam).\tAgain tending to lactic sourness.\tSame for the taste. With some yeast and banana.", "user/profileName": "stcules", "review/aroma": 2.0},
{"review/appearance": 3.0, "beer/style": "English Strong Ale", "review/palate": 3.0, "review/taste": 3.0, "beer/name": "Red Moon", "review/timeUnix": 1235915097, "beer/ABV": 6.2, "beer/beerId": "48213", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 1, "hour": 13, "min": 44, "sec": 57, "mon": 3, "year": 2009, "yday": 60, "wday": 6}, "review/overall": 3.0, "review/text": "Dark red color, light beige foam, average.\tIn the smell malt and caramel, not really light.\tAgain malt and caramel in the taste, not bad in the end.\tMaybe a note of honey in teh back, and a light fruitiness.\tAverage body.\tIn the aftertaste a light bitterness, with the malt and red fruit.\tNothing exceptional, but not bad, drinkable beer.", "user/profileName": "stcules", "review/aroma": 2.5},
{"review/appearance": 3.0, "beer/style": "Foreign / Export Stout", "review/palate": 3.0, "review/taste": 3.0, "beer/name": "Black Horse Black Beer", "review/timeUnix": 1235916604, "beer/ABV": 6.5, "beer/beerId": "48215", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 1, "hour": 14, "min": 10, "sec": 4, "mon": 3, "year": 2009, "yday": 60, "wday": 6}, "review/overall": 3.0, "review/text": "Almost totally black. Beige foam, quite compact, not bad.\tLight smell, just a bit of roast, and some hop. A bit too light.\tThe taste is light oo, and drinkable, with some malt, roast, hints of coffee.\tNothing exceptional, but after all drinkable and pleasant.\tLight to average body.\tIn the aftertaste some dust, somr roast, hint of caramel, and a bit of bitterness.\tNo defect, drinkable, not bad.", "user/profileName": "stcules", "review/aroma": 2.5},
{"review/appearance": 3.5, "beer/style": "German Pilsener", "review/palate": 2.5, "review/taste": 3.0, "beer/name": "Sausa Pils", "review/timeUnix": 1234725145, "beer/ABV": 5.0, "beer/beerId": "47969", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 15, "hour": 19, "min": 12, "sec": 25, "mon": 2, "year": 2009, "yday": 46, "wday": 6}, "review/overall": 3.0, "review/text": "Golden yellow color. White, compact foam, quite creamy. Good appearance.\tFresh smell, with good hop. Quite dry, with a good grassy note. Hay. Fresh and pleasant.\tMore sweet in the mouth, with honey. The hop comes back in the end, and in the aftertaste.\tNot bad, but a bit too sweet for a pils.\tIn the end some vanilla and camomile note.\tIn the aftertaste, too. Though the hop, a bit too sweet.\tHonest.", "user/profileName": "stcules", "review/aroma": 3.0},
{"review/appearance": 4.0, "beer/style": "American Double / Imperial IPA", "review/palate": 4.0, "review/taste": 4.5, "beer/name": "Cauldron DIPA", "review/timeUnix": 1293735206, "user/gender": "Male", "user/birthdayRaw": "Jun 16, 1901", "beer/ABV": 7.7, "beer/beerId": "64883", "user/birthdayUnix": -2163081600, "beer/brewerId": "1075", "review/timeStruct": {"isdst": 0, "mday": 30, "hour": 18, "min": 53, "sec": 26, "mon": 12, "year": 2010, "yday": 364, "wday": 3}, "user/ageInSeconds": 3581417047, "review/overall": 4.0, "review/text": "According to the website, the style for the Caldera Cauldron changes every year. The current release is a DIPA, which frankly is the only cauldron I'm familiar with (it was an IPA/DIPA the last time I ordered a cauldron at the horsebrass several years back). In any event... at the Horse Brass yesterday.\t\tThe beer pours an orange copper color with good head retention and lacing. The nose is all hoppy IPA goodness, showcasing a huge aroma of dry citrus, pine and sandlewood. The flavor profile replicates the nose pretty closely in this West Coast all the way DIPA. This DIPA is not for the faint of heart and is a bit much even for a hophead like myslf. The finish is quite dry and hoppy, and there's barely enough sweet malt to balance and hold up the avalanche of hoppy bitterness in this beer. Mouthfeel is actually fairly light, with a long, persistentely bitter finish. Drinkability is good, with the alcohol barely noticeable in this well crafted beer. Still, this beer is so hugely hoppy/bitter, it's really hard for me to imagine ordering more than a single glass. Regardless, this is a very impressive beer from the folks at Caldera.", "user/profileName": "johnmichaelsen", "review/aroma": 4.5}]

Answer 1

Use ast.literal_eval to read lines in to python dicts and convert to a valid json using json.dumps

import json
import ast

with open("./test.json", "r") as f:
    for line in f:
        print(json.dumps(ast.literal_eval(line)))

Output:

{"review/appearance": 2.5, "beer/style": "Hefeweizen", "review/palate": 1.5, "review/taste": 1.5, "beer/name": "Sausa Weizen", "review/timeUnix": 1234817823, "beer/ABV": 5.0, "beer/beerId": "47986", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 16, "hour": 20, "min": 57, "sec": 3, "mon": 2, "year": 2009, "yday": 47, "wday": 0}, "review/overall": 1.5, "review/text": "A lot of foam. But a lot.\tIn the smell some banana, and then lactic and tart. Not a good start.\tQuite dark orange in color, with a lively carbonation (now visible, under the foam).\tAgain tending to lactic sourness.\tSame for the taste. With some yeast and banana.", "user/profileName": "stcules", "review/aroma": 2.0}
{"review/appearance": 3.0, "beer/style": "English Strong Ale", "review/palate": 3.0, "review/taste": 3.0, "beer/name": "Red Moon", "review/timeUnix": 1235915097, "beer/ABV": 6.2, "beer/beerId": "48213", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 1, "hour": 13, "min": 44, "sec": 57, "mon": 3, "year": 2009, "yday": 60, "wday": 6}, "review/overall": 3.0, "review/text": "Dark red color, light beige foam, average.\tIn the smell malt and caramel, not really light.\tAgain malt and caramel in the taste, not bad in the end.\tMaybe a note of honey in teh back, and a light fruitiness.\tAverage body.\tIn the aftertaste a light bitterness, with the malt and red fruit.\tNothing exceptional, but not bad, drinkable beer.", "user/profileName": "stcules", "review/aroma": 2.5}
{"review/appearance": 3.0, "beer/style": "Foreign / Export Stout", "review/palate": 3.0, "review/taste": 3.0, "beer/name": "Black Horse Black Beer", "review/timeUnix": 1235916604, "beer/ABV": 6.5, "beer/beerId": "48215", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 1, "hour": 14, "min": 10, "sec": 4, "mon": 3, "year": 2009, "yday": 60, "wday": 6}, "review/overall": 3.0, "review/text": "Almost totally black. Beige foam, quite compact, not bad.\tLight smell, just a bit of roast, and some hop. A bit too light.\tThe taste is light oo, and drinkable, with some malt, roast, hints of coffee.\tNothing exceptional, but after all drinkable and pleasant.\tLight to average body.\tIn the aftertaste some dust, somr roast, hint of caramel, and a bit of bitterness.\tNo defect, drinkable, not bad.", "user/profileName": "stcules", "review/aroma": 2.5}
{"review/appearance": 3.5, "beer/style": "German Pilsener", "review/palate": 2.5, "review/taste": 3.0, "beer/name": "Sausa Pils", "review/timeUnix": 1234725145, "beer/ABV": 5.0, "beer/beerId": "47969", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 15, "hour": 19, "min": 12, "sec": 25, "mon": 2, "year": 2009, "yday": 46, "wday": 6}, "review/overall": 3.0, "review/text": "Golden yellow color. White, compact foam, quite creamy. Good appearance.\tFresh smell, with good hop. Quite dry, with a good grassy note. Hay. Fresh and pleasant.\tMore sweet in the mouth, with honey. The hop comes back in the end, and in the aftertaste.\tNot bad, but a bit too sweet for a pils.\tIn the end some vanilla and camomile note.\tIn the aftertaste, too. Though the hop, a bit too sweet.\tHonest.", "user/profileName": "stcules", "review/aroma": 3.0}
{"review/appearance": 4.0, "beer/style": "American Double / Imperial IPA", "review/palate": 4.0, "review/taste": 4.5, "beer/name": "Cauldron DIPA", "review/timeUnix": 1293735206, "user/gender": "Male", "user/birthdayRaw": "Jun 16, 1901", "beer/ABV": 7.7, "beer/beerId": "64883", "user/birthdayUnix": -2163081600, "beer/brewerId": "1075", "review/timeStruct": {"isdst": 0, "mday": 30, "hour": 18, "min": 53, "sec": 26, "mon": 12, "year": 2010, "yday": 364, "wday": 3}, "user/ageInSeconds": 3581417047, "review/overall": 4.0, "review/text": "According to the website, the style for the Caldera Cauldron changes every year. The current release is a DIPA, which frankly is the only cauldron I'm familiar with (it was an IPA/DIPA the last time I ordered a cauldron at the horsebrass several years back). In any event... at the Horse Brass yesterday.\t\tThe beer pours an orange copper color with good head retention and lacing. The nose is all hoppy IPA goodness, showcasing a huge aroma of dry citrus, pine and sandlewood. The flavor profile replicates the nose pretty closely in this West Coast all the way DIPA. This DIPA is not for the faint of heart and is a bit much even for a hophead like myslf. The finish is quite dry and hoppy, and there's barely enough sweet malt to balance and hold up the avalanche of hoppy bitterness in this beer. Mouthfeel is actually fairly light, with a long, persistentely bitter finish. Drinkability is good, with the alcohol barely noticeable in this well crafted beer. Still, this beer is so hugely hoppy/bitter, it's really hard for me to imagine ordering more than a single glass. Regardless, this is a very impressive beer from the folks at Caldera.", "user/profileName": "johnmichaelsen", "review/aroma": 4.5}

EDIT: (Code for the updated question)

output = []
with open("./test.json", "r") as f:
    for line in f:
        output.append(ast.literal_eval(line))
print(json.dumps(output))

Output:

[{"review/appearance": 2.5, "beer/style": "Hefeweizen", "review/palate": 1.5, "review/taste": 1.5, "beer/name": "Sausa Weizen", "review/timeUnix": 1234817823, "beer/ABV": 5.0, "beer/beerId": "47986", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 16, "hour": 20, "min": 57, "sec": 3, "mon": 2, "year": 2009, "yday": 47, "wday": 0}, "review/overall": 1.5, "review/text": "A lot of foam. But a lot.\tIn the smell some banana, and then lactic and tart. Not a good start.\tQuite dark orange in color, with a lively carbonation (now visible, under the foam).\tAgain tending to lactic sourness.\tSame for the taste. With some yeast and banana.", "user/profileName": "stcules", "review/aroma": 2.0}, {"review/appearance": 3.0, "beer/style": "English Strong Ale", "review/palate": 3.0, "review/taste": 3.0, "beer/name": "Red Moon", "review/timeUnix": 1235915097, "beer/ABV": 6.2, "beer/beerId": "48213", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 1, "hour": 13, "min": 44, "sec": 57, "mon": 3, "year": 2009, "yday": 60, "wday": 6}, "review/overall": 3.0, "review/text": "Dark red color, light beige foam, average.\tIn the smell malt and caramel, not really light.\tAgain malt and caramel in the taste, not bad in the end.\tMaybe a note of honey in teh back, and a light fruitiness.\tAverage body.\tIn the aftertaste a light bitterness, with the malt and red fruit.\tNothing exceptional, but not bad, drinkable beer.", "user/profileName": "stcules", "review/aroma": 2.5}, {"review/appearance": 3.0, "beer/style": "Foreign / Export Stout", "review/palate": 3.0, "review/taste": 3.0, "beer/name": "Black Horse Black Beer", "review/timeUnix": 1235916604, "beer/ABV": 6.5, "beer/beerId": "48215", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 1, "hour": 14, "min": 10, "sec": 4, "mon": 3, "year": 2009, "yday": 60, "wday": 6}, "review/overall": 3.0, "review/text": "Almost totally black. Beige foam, quite compact, not bad.\tLight smell, just a bit of roast, and some hop. A bit too light.\tThe taste is light oo, and drinkable, with some malt, roast, hints of coffee.\tNothing exceptional, but after all drinkable and pleasant.\tLight to average body.\tIn the aftertaste some dust, somr roast, hint of caramel, and a bit of bitterness.\tNo defect, drinkable, not bad.", "user/profileName": "stcules", "review/aroma": 2.5}, {"review/appearance": 3.5, "beer/style": "German Pilsener", "review/palate": 2.5, "review/taste": 3.0, "beer/name": "Sausa Pils", "review/timeUnix": 1234725145, "beer/ABV": 5.0, "beer/beerId": "47969", "beer/brewerId": "10325", "review/timeStruct": {"isdst": 0, "mday": 15, "hour": 19, "min": 12, "sec": 25, "mon": 2, "year": 2009, "yday": 46, "wday": 6}, "review/overall": 3.0, "review/text": "Golden yellow color. White, compact foam, quite creamy. Good appearance.\tFresh smell, with good hop. Quite dry, with a good grassy note. Hay. Fresh and pleasant.\tMore sweet in the mouth, with honey. The hop comes back in the end, and in the aftertaste.\tNot bad, but a bit too sweet for a pils.\tIn the end some vanilla and camomile note.\tIn the aftertaste, too. Though the hop, a bit too sweet.\tHonest.", "user/profileName": "stcules", "review/aroma": 3.0}, {"review/appearance": 4.0, "beer/style": "American Double / Imperial IPA", "review/palate": 4.0, "review/taste": 4.5, "beer/name": "Cauldron DIPA", "review/timeUnix": 1293735206, "user/gender": "Male", "user/birthdayRaw": "Jun 16, 1901", "beer/ABV": 7.7, "beer/beerId": "64883", "user/birthdayUnix": -2163081600, "beer/brewerId": "1075", "review/timeStruct": {"isdst": 0, "mday": 30, "hour": 18, "min": 53, "sec": 26, "mon": 12, "year": 2010, "yday": 364, "wday": 3}, "user/ageInSeconds": 3581417047, "review/overall": 4.0, "review/text": "According to the website, the style for the Caldera Cauldron changes every year. The current release is a DIPA, which frankly is the only cauldron I'm familiar with (it was an IPA/DIPA the last time I ordered a cauldron at the horsebrass several years back). In any event... at the Horse Brass yesterday.\t\tThe beer pours an orange copper color with good head retention and lacing. The nose is all hoppy IPA goodness, showcasing a huge aroma of dry citrus, pine and sandlewood. The flavor profile replicates the nose pretty closely in this West Coast all the way DIPA. This DIPA is not for the faint of heart and is a bit much even for a hophead like myslf. The finish is quite dry and hoppy, and there's barely enough sweet malt to balance and hold up the avalanche of hoppy bitterness in this beer. Mouthfeel is actually fairly light, with a long, persistentely bitter finish. Drinkability is good, with the alcohol barely noticeable in this well crafted beer. Still, this beer is so hugely hoppy/bitter, it's really hard for me to imagine ordering more than a single glass. Regardless, this is a very impressive beer from the folks at Caldera.", "user/profileName": "johnmichaelsen", "review/aroma": 4.5}]

Output is a valid json, you can check here: https://jsonlint.com/ .

Answer 2

Why don't you just load it in the first place and then do whatever you want to it? something like this

import json

with open('./test.json','r') as f:
    data = json.load(f)
    s = data.replace('\t','')
    s = s.replace('\n','')
    s = s.replace("\'", "\"")
    
    print(s)

Answer 3

The issue is that you're using json.load on an str object ( s ). load expects a file object that supports read. Notice that s is a string, the result of f.read .

If you want to use json.load , you need to feed f into it. If you want to load the json from a string, use json.loads

data = json.loads(s)
print(data)

You were also trying to print s for whatever reason. Even though the result json is in data .