Converting a json website into a dictionary in python

Question

I'm trying to write a function to fetch data from the website below into a dictionary in python and I can't seem to find a way to do it anywhere, if anyone could give me a point in the right direction that'd be great.

http://openexchangerates.org/api/currencies.json

What I have so far is this, but I keep getting an error saying 'The JSON object must be str, not bytes'

import urllib, json
url = 'http://openexchangerates.org/api/currencies.json'
response = urllib.request.urlopen(url);
data = json.loads(response.read())
print(data)

Answer 1

You need to decode your JSON data to a string:

import urllib.request
import json

url = 'http://openexchangerates.org/api/currencies.json'
response = urllib.request.urlopen(url)
encoding = response.info().get_content_charset('utf8')
data = json.loads(response.read().decode(encoding))

This'll use the correct codec as specified by the HTTP server, or UTF-8 if none was specified. UTF-8 is the default codec for JSON data.

In this case, the server has explicitly set the character set to UTF-8:

>>> import urllib.request
>>> import json
>>> url = 'http://openexchangerates.org/api/currencies.json'
>>> response = urllib.request.urlopen(url)
>>> response.info().get('content-type')
'application/json; charset=utf-8'
>>> response.info().get_content_charset()
'utf-8'

so decoding with that codec then produces the expected input for json.loads() :

>>> encoding = response.info().get_content_charset('utf8')
>>> data = json.loads(response.read().decode(encoding))
>>> data.keys()
dict_keys(['BYR', 'VUV', 'TTD', 'PHP', 'CRC', 'KWD', 'BGN', 'XDR', 'PGK', 'STD', 'MZN', 'XCD', 'ANG', 'MTL', 'AZN', 'LTL', 'MXN', 'TJS', 'CVE', 'DZD', 'MYR', 'MVR', 'GTQ', 'IQD', 'TRY', 'ERN', 'TOP', 'TWD', 'LYD', 'BDT', 'INR', 'CLF', 'ZWL', 'XPF', 'MOP', 'LKR', 'PLN', 'IRR', 'HRK', 'GYD', 'DOP', 'UGX', 'BZD', 'IDR', 'AWG', 'GHS', 'IMP', 'BND', 'MGA', 'DJF', 'ARS', 'MWK', 'COP', 'DKK', 'UYU', 'HUF', 'NOK', 'AED', 'XAG', 'LBP', 'KHR', 'MUR', 'HTG', 'AUD', 'LVL', 'RON', 'ETB', 'ISK', 'NZD', 'RSD', 'THB', 'SYP', 'RWF', 'VND', 'CAD', 'MMK', 'GIP', 'BOB', 'MAD', 'KES', 'SAR', 'BTN', 'WST', 'NIO', 'GNF', 'BHD', 'NGN', 'UZS', 'USD', 'GEL', 'BWP', 'CZK', 'SVC', 'FJD', 'JEP', 'CLP', 'EEK', 'BBD', 'JPY', 'BSD', 'CUP', 'SCR', 'XAU', 'TND', 'YER', 'AMD', 'PKR', 'GGP', 'SLL', 'CUC', 'PAB', 'LRD', 'SEK', 'BRL', 'MDL', 'ZMW', 'ALL', 'SOS', 'TZS', 'BAM', 'CDF', 'KRW', 'SDG', 'UAH', 'KYD', 'FKP', 'XAF', 'SGD', 'VEF', 'LAK', 'QAR', 'MKD', 'PEN', 'CHF', 'MRO', 'ILS', 'AFN', 'ZAR', 'SHP', 'HNL', 'KMF', 'XOF', 'HKD', 'GBP', 'EGP', 'BMD', 'AOA', 'PYG', 'SRD', 'BTC', 'EUR', 'JOD', 'NPR', 'KZT', 'CNY', 'KGS', 'ZMK', 'LSL', 'GMD', 'MNT', 'OMR', 'BIF', 'NAD', 'SZL', 'JMD', 'SBD', 'KPW', 'RUB', 'TMT'])

Answer 2

Quick fix - decode the raw bytes you get from http to text, using its decode method:

import urllib, json
url = 'http://openexchangerates.org/api/currencies.json'
response = urllib.request.urlopen(url);
data = json.loads(response.read().decode("utf-8"))
print(data)

Long term fix: understand what text data (and unicode) is, as compared to actual bytes, which are numbers in the 0-256 range that are actually sent over network protocols, stored on disk and databases, etc...often representing text. To that end, I strongly recomend you read this article:

http://www.joelonsoftware.com/articles/Unicode.html