Python ValueError: No JSON Object Could Be Decoded (Intermittent)

Question

Getting an error with the following code. However, sometimes it works and sometimes it doesn't. I have validated the data with JSONLint. I have tried to decode/encode the data based off responses to similar error questions and also tried the same using the codecs module. What I don't understand is why it sometimes works fine and then other times throws an error.

import requests
import json
import codecs

response = requests.get("https://api.weather.gov/alerts/active")
print(response.status_code)

with open ('data.json', 'w') as file:
    file.write(response.content)

myFile = open('data.json', 'r')
myObject = myFile.read()
u = myObject.decode('utf-8-sig')
myObject = u.encode('utf-8')
myFile.encoding
myFile.close()
myData = json.loads(myObject, 'utf-8') 

#myData=json.loads(codecs.open(myObject,'r', 'utf-8'))

A sample of the JSON data can be found in another question I asked here .

Answer 1

The problem is that your JSON actually is invalid.

Taking the example from your other question , you can paste it into a JSON validator like this one and see the problem: On line 35, there's a trailing comma after the last value in an array. That's not legal JSON . So no conforming JSON parser is going to parse it.

This has nothing to do with character encoding. So throwing in a bunch of extra encode and decode calls and pulling in extra libraries to do it even more times isn't going to help. You've decoded the bytes properly to text, but that text isn't valid JSON.

---

Why does it happen sometimes but not other times? Presumably (just based on the name of the endpoint) that API is gathering alerts from various places on their backend, and sometimes one of those alerts is encoded incorrectly, but most of the time they're all fine (or there aren't any). Who knows? That's not your bug to fix; it's theirs.

In fact, the first thing you should do is file a bug against the provider, showing them the bad JSON you got, and pointing out exactly what's wrong with it. (If you have some kind of support contract with them, you may want to ask them to do something about it so you don't have to hack around their problems, but I assume you don't.)

---

What can you do in the mean time? Well, if you can just try / except JSONDecodeError: and skip the occasional bad value, that's probably the best answer until they fix things. If having incomplete data invalidates your whole program, however, that's no good. In particular, in this case, it might mean you get no data for the entire time some important alert was up, so you never see it.

There are "sloppy JSON" decoders out there that you can try. This is generally a bad idea, because it's easy to end up silently decoding something incorrectly in the wrong way, and getting garbage data into your collection. Which also means that most such projects get abandoned, once the author decides it was a bad idea. But people keep writing new ones, so if you really want to, you can probably search for and find one.

If the errors in their JSON are consistent , and you can work out how to describe them, you should be able to write a special-purpose "sloppy JSON" decoder that handles exactly those errors, or a preprocessor that fixes those errors before decoding. This will be non-trivial and probably a bit ugly, but then maybe it should be ugly—every time you read it, it'll remind you to check whether the provider has fixed their bug so you can throw out that ugly code. At any rate, nobody can work out whether their errors are consistent, and then write code that deals with them, without a lot more than just a single example. And probably nobody wants to go through a whole mess of invalid JSON for you anyway. So if you want to go this way, you're going to have to try to do this yourself (but of course you can always ask a new question if you get stuck, with all the relevant details in it).

Answer 2

Why do you keep the code so complicated? You can just do:

obj = requests.get("https://api.weather.gov/alerts/active").json()

The json object is automatically decoded for you into a pythonic dict . It seems to be working consistently with python 2.7.14 and 3.6.5 .