importing json file from github into python. Getting Error: JSONDecodeError: Expecting value: line 7 column 1 (char 6)

Question

Here is my code:

import re, json, requests

url = 'https://github.com/caminofinancial/data-eng-take-home/blob/master/prequalresult.json'

resp = requests.get(url)
resp_parsed = re.sub(r'^jsonp\d+\(|\)\s+$', '', resp.text)
data = json.loads(resp_parsed)
print(data)

And I Got the error: JSONDecodeError: Expecting value: line 7 column 1 (char 6). Can someone check it and solve the issue?

Answer 1

Use the raw GitHub URL when you need to access the file directly. You can get it by clicking the 'Raw' button on the page.

 url = 'https://raw.githubusercontent.com/caminofinancial/data-eng-take-home/master/prequalresult.json'
 resp = requests.get(url)
 data = json.loads(resp.text)
 print(data)

Answer 2

from pyspark import SparkFiles

zip_url = "https://raw.githubusercontent.com/spark-examples/spark-scala-examples/master/src/main/resources/zipcodes.json"

spark.sparkContext.addFile(zip_url)

zip_df = spark.read.json("file://" +SparkFiles.get("zipcodes.json"))

#click on raw and then copy url