Convert a list of dicts to DataFrame and back to exactly same list?

Question

In my Django view, I want to load a list of dictionaries into a Pandas dataframe, manipulate it, dump to dict again and return such manipulated data as a part of this view's JSON response:

def test(request):
    pre_pandas = [{'type': 'indoor', 'speed': 1, 'heart_rate': None}, {'type': 'outdoor', 'speed': 2, 'heart_rate': 124.0}, {'type': 'commute', 'speed': 3, 'heart_rate': 666.0}, {'type': 'indoor', 'speed': 4, 'heart_rate': 46.0}]

    df = pd.DataFrame(pre_pandas)

    # some data manipulation here...

    post_pandas = df.to_dict(orient='records')

    response = {
        'pre_pandas': pre_pandas,
        'post_pandas': post_pandas,
    }

    return JsonResponse(response)

The problem with my approach is that Pandas' to_dict() method replaces Python's None with nan , so that the response has NaN in it:

 {"pre_pandas": [{"type": "indoor", "speed": 1, "heart_rate": null}...], "post_pandas": [{"heart_rate": NaN, "speed": 1,...}]

and JavaScript cannot tackle NaN .

Is there a way to dump a dataframe to a dict so that the output is exactly the same as the dict it was build from?

I could possibly adjust the data manually with a list replace() method, but it feels awkward and also I would need to cover for all of the other - if any - conversions Pandas' to_dict() method might do.

I also cannot dump post_pandas to JSON, as I am already doing it in JsonResponse .

Answer 1

Your the columns in your pre_pandas Dataframe are inferred from the provided dictionary, to prevent this you can explicitly specify the data type object. What forces all the values to be of type object.

Like that:

df = pd.DataFrame(pre_pandas, dtype=object)