pandas DataFrame to list of dicts using to_json()

Question

So, i'm reading a xlsx file with pandas, then parsing the datetime (excel's a float)

Then I need to parse it into Json, and I'm running into some problems.

STEP 1 (Before parsing with to_json())

df = pandas.read_excel('test.xlsx', names=['date', 'value', 'source'])
df['date'] = pandas.to_datetime(df['date'], format='%b %d %Y.%f')
print(df)

the return is

        date  value                            source
0 2012-05-22      1              xxxxxxxxxxxxxxxxxxxx
1 2012-05-25      1                     xxxxxxxxxxxxx
2 2012-05-30      1  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
3 2012-06-01      1                        xxxxxxxxxx
4 2012-06-08      1                 xxxxxxxxxxxxxxxxx

Everything seems fine, then I got o to_json

payload = df.to_json()

And the return is

{"date":{"0":1337644800000,"1":1337904000000,"2":1338336000000,"3":1338508800000,"4":1339113600000},"value":{"0":1,"1":1,"2":1,"3":1,"4":1},"source":{"0":"xxxxxxxxxxxxxxx","1":"xxxxxxxxxx","2":"xxxxxxxxxxx","3":"xxxxxxxxxxxxxxxx","4":"xxxxxxxxxxxxxxx"}}

So what am I doing wrong? Am I missing args on to_json()? Halp pls :c

I needed it to be like this:

[{"date":"2012-05-22","value":1,"source":"xxxxxxxxxxxxxxxxxxxx"},
{"date":"2012-05-25","value":1,"source":"xxxxxxxxxxxxxxxxxxxx"},
{"date":"2012-05-30","value":1,"source":"xxxxxxxxxxxxxxxxxxxxx"},
{"date":"2012-06-01","value":1,"source":"xxxxxxxxxxxxxxxxxxxx"},
{"date":"2012-06-08","value":1,"source":"xxxxxxxxxxxxxxxxxxxxxx"}]

Answer 1

You'll need a couple of fixes—

1. Convert your date column to string, because as it currently is, your datetime column is being coerced to Unix integer timestamps. Alternatively, use the date_format argument with to_json as the other answer suggests.
2. Change the orient when you save to json; specify orient='records' .

df['date'] = pandas.to_datetime(df['date'], format='%b %d %Y.%f').astype(str)
payload = df.to_json(orient='records')

print(payload)
'[{"date":"2012-05-22","source":"xxxxxxxxxxxxxxxxxxxx","value":1},{"date":"2012-05-25","source":"xxxxxxxxxxxxxxxxxxxx","value":1},{"date":"2012-05-30","source":"xxxxxxxxxxxxxxxxxxxxx","value":1},{"date":"2012-06-01","source":"xxxxxxxxxxxxxxxxxxxx","value":1},{"date":"2012-06-08","source":"xxxxxxxxxxxxxxxxxxxxxx","value":1}]'

Answer 2

Although if you convert to string (as mentioned in other questions and comments) you can get the format you want, you may not need to. Check out the to_json() parameter date_format . I believe you want .to_json(..., date_format='iso') .

As per the documentation for the date_format parameter:

For orient='table', the default is 'iso'. For all other orients, the default is 'epoch'.