How to create a pandas dataframe from two dictionaries with same key values?

Question

Say I have two dictionaries of the following form:

{'A':[1,2,3,4,5,6,7],
 'B':[12,13,14,15,16,17,18} - Belongs to category "M"

{'A':[8,9,10,11,12,13,14],
 'B':[18,19,20,21,22,23,24]} - Belongs to category "P"

Now the resulting data frame should be of the form --

Name . Value . Category

A    .  1    .  M
A    .  8    .  P
A    .  10   .  P
B    .  12   .  M

And so on. How can something like this be achieved?

Answer 1

Solution

Here is a more pandorable approach than what is suggested by user3483203. This avoids unnecessary iteration, is faster (for big enough datasets), and more idiomatic.

m = {'A':[1,2,3,4,5,6,7],
     'B':[12,13,14,15,16,17,18]}

p = {'A':[8,9,10,11,12,13,14],
     'B':[18,19,20,21,22,23,24]}


p_df = pd.DataFrame(p).melt(value_name='value')
m_df = pd.DataFrame(m).melt(value_name='value')

p_df['category'] = 'P'
m_df['category'] = 'M'

result = pd.concat([m_df, p_df], ignore_index=True)

---

Benchmarks (for bigger datasets):

m = {'A': list(range(0, 100_000)), 'B': list(range(100_000, 200_000))}
p = {'A': list(range(200_000, 300_000)), 'B': list(range(300_000, 400_000))}

Here we go:

%%timeit
p_df = pd.DataFrame(p).melt(value_name='value')
m_df = pd.DataFrame(m).melt(value_name='value')

p_df['category'] = 'P'
m_df['category'] = 'M'

result = pd.concat([m_df, p_df], ignore_index=True)

120 ms ± 3.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
categories = ['M', 'P']
dcts = [m, p]
dfs = [
    pd.DataFrame([[k, el, cat] for k, v in dct.items() for el in v])
    for dct, cat in zip(dcts, categories)
]

cols = {'columns': {0: 'Name', 1: 'Value', 2: 'Category'}}
result = pd.concat(dfs).reset_index(drop=True).rename(**cols)

207 ms ± 8.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Answer 2

Setup

d1 = {'A': [1, 2, 3, 4, 5, 6, 7], 'B': [12, 13, 14, 15, 16, 17, 18]}
d2 = {'A': [8, 9, 10, 11, 12, 13, 14], 'B': [18, 19, 20, 21, 22, 23, 24]}

categories = ['M', 'P']
dcts = [d1, d2]

---

Assuming that you know which category goes with which dictionary, you can restructure your dictionaries and use concat :

dfs = [
    pd.DataFrame([[k, el, cat] for k, v in dct.items() for el in v])
    for dct, cat in zip(dcts, categories)
]

cols = {'columns': {0: 'Name', 1: 'Value', 2: 'Category'}}
pd.concat(dfs).reset_index(drop=True).rename(**cols)

   Name  Value Category
0     A      1        M
1     A      2        M
2     A      3        M
3     A      4        M
4     A      5        M
5     A      6        M
6     A      7        M
7     B     12        M
8     B     13        M
9     B     14        M
10    B     15        M
11    B     16        M
12    B     17        M
13    B     18        M
14    A      8        P
15    A      9        P
16    A     10        P
17    A     11        P
18    A     12        P
19    A     13        P
20    A     14        P
21    B     18        P
22    B     19        P
23    B     20        P
24    B     21        P
25    B     22        P
26    B     23        P
27    B     24        P