如何从存储在多个嵌套字典中的数据创建 Pandas 框架？

Question

I have a Python program in which I sweep multiple parameters and at each point I calculate a few results. I then want to export the results in the form of a CSV (or Excel) report that, on each row, contains the parameters and results. For example, here I sweep two parameters i and j and calculated res1 and res2 as a function of i and j . (This is completely silly MWE though!)

res1 = dict()
res2 = dict()

for i in range(5):
    res1[i] = dict()
    res2[i] = dict()
    for j in range(5):
        res1[i][j] = i+j
        res2[i][j] = i*j

And I would like to create a CSV with 25 rows and 4 columns where first two columns are (i, j) combinations for which res1 and res2 are calculated and second two columns are res1 and res2 respectively. A naive way of exporting such a CSV is as follows:

#### Naive CSV writing
print(', '.join(['i', 'j', 'res1', 'res2']))

for i in range(5):
    for j in range(5):
        print(', '.join([str(i), str(j), str(res1[i][j]), str(res2[i][j])]))

I was wondering if there is a way to create a pandas frame from the dictionaries so that then I can export the reports more easily?

I know that pandas.DataFrame constructor accepts a dictionary that maps column headers to column values. So, for example the following is a possible solution:

import pandas as pd
import sys

# generate results as before 

d = dict([('i', list()),
          ('j', list()),
          ('res1', list()),
          ('res2', list())])

for i in range(5):
    for j in range(5):
        d['i'].append(i)
        d['j'].append(j)
        d['res1'].append(res1[i][j])
        d['res2'].append(res2[i][j])

df = pd.DataFrame(data=d)

df.to_csv(sys.stdout, index=False)

Yet, the above does not look so elegant (and I think is not efficient either). Is there a better way to do so?

Answer 1

You could create normal list

data = []
for i in range(5):
    for j in range(5):
        data.append([i, j, res1[i][j], res2[i][j]])

And then convert to DataFrame

import pandas as pd

df = pd.DataFrame(data, columns=['i', 'j', 'res1', 'res2'])

print(df)

Or directly write it using csv module

import csv

fh = open("output.csv", 'w')
csvwriter = cvs.writer(fh)

csvwriter.writerow(['i', 'j', 'res1', 'res2'])

for i in range(5):
    for j in range(5):
        csvwriter.writerow([i, j, res1[i][j], res2[i][j]])

fh.close()

Answer 2

How about this:

import pandas as pd
from itertools import product
p = np.array(list(product(range(5), range(5))))

df = pd.DataFrame(data={'i': p[:,0], 'j':p[:,1]})

def res(row):
    row['res1'] = res1(row['i'], row['j'])
    row['res2'] = res2(row['i'], row['j'])

    return row

df = df.apply(res, axis=1)

Now you can write the dataframe directly into a csv