ValueError：DataFrame 构造函数未正确调用！ 当将列表中的字典覆盖到 pandas dataframe

Question

I want to convert a list of dictionaries to a pandas dataframe, however, I got ValueError: DataFrame constructor not properly called!

Below is an example and how I got the data:

import requests
import pandas as pd

# Send an HTTP GET request to the URL
response = requests.get(url)

# Decode the JSON data into a dictionary
scrapped_data = response.text

Content of response.text is:

[{"id":123456,"date":"12-12-2022","value":37},{"id":123456,"date":"13-12-2022","value":38}]

I want to convert it to a dataframe format like the following:

id	date	value
123456	12-12-2022	37
123456	13-12-2022	38

I tried the following methods:

df = pd.DataFrame(scrapped_data)

df = pd.DataFrame_from_dict(scrapped_data)

df = pd.DataFrame(scrapped_data, orient='columns')

all got the same value errors.

I also tried:

df = pd.json_normalize(scrapped_data)

but got NotImplementedError

The type for scrapped_data is string format

Thanks for your help, let me know if you have any questions

Answer 1

As you said, scrapped_data is a string then you need to convert it into a dictionary (with the method loads from the json library for example).

If scrapped_data = '[{"id":"123456","date":"12-12-2022","value":"37"}, {"id":"123456","date":"13-12-2022","value":"38"}]', then you can just do df = pd.DataFrame(scrapped_data).

Answer 2

One reason for receiving this error from pandas is providing str as data. I think your data come as str , If it is the case then Try this:

import json
import pandas as pd

orignal_data='[{"id":"123456","date":"12-12-2022","value":"37"}, {"id":"123456","date":"13-12-2022","value":"38"}]'
scraped_data = json.loads(orignal_data)
df = pd.DataFrame(data=scraped_data)
df