Consume REStful API data with python => from JSON to pandas DataFrame

Question

from flask import Flask, request
from flask_restful import Resource, Api, reqparse
#from flask_jwt import JWT, jwt_required
from pymongo import MongoClient
import pandas as pd

app = Flask(__name__)
app.secret_key = 'xxx'
api = Api(app)

class Data(Resource):
    def get(self):
        client = MongoClient("localhost", 27017)
        db = client.test
        collection = db.my_collection.find({})
        df = pd.DataFrame(list(collection))
        df = df.iloc[1:5:,1:(len(df.columns))] ### ID Variable raushauen, führt zu OverflowError
        return df.to_json(orient="columns")

api.add_resource(Data, "/data/") # http://127.0.0.1:5000/data/

app.run(port = 5000, debug=True)

Hello everybody. This is my little API which runs on localhost. I just send a subset of the titanic dataset.

When I try to access the data in the browser I receive the following:

"{\"PassengerId\":{\"1\":2,\"2\":3,\"3\":4,\"4\":5},\"Survived\":{\"1\":1,\"2\":1,\"3\":1,\"4\":0},\"Pclass\":{\"1\":1,\"2\":3,\"3\":1,\"4\":3},\"Name\":{\"1\":\"Cumings, Mrs. John Bradley (Florence Briggs Thayer)\",\"2\":\"Heikkinen, Miss. Laina\",\"3\":\"Futrelle, Mrs. Jacques Heath (Lily May Peel)\",\"4\":\"Allen, Mr. William Henry\"},\"Sex\":{\"1\":\"female\",\"2\":\"female\",\"3\":\"female\",\"4\":\"male\"},\"Age\":{\"1\":38.0,\"2\":26.0,\"3\":35.0,\"4\":35.0},\"SibSp\":{\"1\":1,\"2\":0,\"3\":1,\"4\":0},\"Parch\":{\"1\":0,\"2\":0,\"3\":0,\"4\":0},\"Ticket\":{\"1\":\"PC 17599\",\"2\":\"STON\\/O2. 3101282\",\"3\":\"113803\",\"4\":\"373450\"},\"Fare\":{\"1\":71.2833,\"2\":7.925,\"3\":53.1,\"4\":8.05},\"Cabin\":{\"1\":\"C85\",\"2\":null,\"3\":\"C123\",\"4\":null},\"Embarked\":{\"1\":\"C\",\"2\":\"S\",\"3\":\"S\",\"4\":\"S\"}}"

Now I try to consume the API and read the data into a dataframe again. The following code results in a strange looking text Object:

r = requests.get('http://localhost:5000/data/')
data = r.text

Out[90]: '"{\\"PassengerId\\":{\\"1\\":2,\\"2\\":3,\\"3\\":4,\\"4\\":5},\\"Survived\\":{\\"1\\":1,\\"2\\":1,\\"3\\":1,\\"4\\":0},\\"Pclass\\":{\\"1\\":1,\\"2\\":3,\\"3\\":1,\\"4\\":3},\\"Name\\":{\\"1\\":\\"Cumings, Mrs. John Bradley (Florence Briggs Thayer)\\",\\"2\\":\\"Heikkinen, Miss. Laina\\",\\"3\\":\\"Futrelle, Mrs. Jacques Heath (Lily May Peel)\\",\\"4\\":\\"Allen, Mr. William Henry\\"},\\"Sex\\":{\\"1\\":\\"female\\",\\"2\\":\\"female\\",\\"3\\":\\"female\\",\\"4\\":\\"male\\"},\\"Age\\":{\\"1\\":38.0,\\"2\\":26.0,\\"3\\":35.0,\\"4\\":35.0},\\"SibSp\\":{\\"1\\":1,\\"2\\":0,\\"3\\":1,\\"4\\":0},\\"Parch\\":{\\"1\\":0,\\"2\\":0,\\"3\\":0,\\"4\\":0},\\"Ticket\\":{\\"1\\":\\"PC 17599\\",\\"2\\":\\"STON\\\\/O2. 3101282\\",\\"3\\":\\"113803\\",\\"4\\":\\"373450\\"},\\"Fare\\":{\\"1\\":71.2833,\\"2\\":7.925,\\"3\\":53.1,\\"4\\":8.05},\\"Cabin\\":{\\"1\\":\\"C85\\",\\"2\\":null,\\"3\\":\\"C123\\",\\"4\\":null},\\"Embarked\\":{\\"1\\":\\"C\\",\\"2\\":\\"S\\",\\"3\\":\\"S\\",\\"4\\":\\"S\\"}}"\n'

which leads to an error when I run the following code:

df = pd.read_json(data, orient="columns")

ValueError: DataFrame constructor not properly called!

This is the first time I create an API... can you tell me in what steps an error occured and how to fix it? Thanks.

Answer 1

It is an issue with your JSON data, try below:

data = r.text
str_data = json.loads(data)
json_data = json.loads(str_data)
pd.DataFrame(json_data)

or

data = r.text
json_data = json.loads(data)
pd.read_json(json_data)