如何用零替换不平衡数据框中的缺失值？

Question

I have a dataframe with with two dimensions, A and B; however this data frame is unbalanced in that some of the values are missing because the database does not include values for all possible combinations of A and B. What I want to do is make sure the dataframe is balanced, and those elements that are missing to be filled with zeros.

I am grabbing the data for the dataframe from an sqlite database that I'm conneting to via SQLAlchemy, using the following code

connection = sqlite3.connect("my_database.db")
cursor = connection.cursor()
# set up a query
cursor.execute('SELECT factorA, factorB,  COUNT(*) as unique_driver_counts FROM block_optimizer_runs GROUP BY 1, 2')
results = cursor.fetchall()
results_df = pd.DataFrame(results, columns=['factorA', 'factorB', 'count'])

But the database doesn't include data for all possible combinations of factorA, factorB, and factorC. When these values do not exist, the database returns no value; but in the dataframe I need these 'missing' values to be filled with a zero.

For example

import pandas as pd
data = [['dog', 'house', 1], ['dog', 'apartment', 2], ['dog', 'trailer', 1], ['dog', 'cabin', 0], ['dog', 'shack', 1],
['cat', 'house', 3], ['cat', 'apartment', 1], ['cat', 'trailer', 0], ['cat', 'shack', 3],
['gecko', 'apartment', 3], ['gecko', 'trailer', 2], ['gecko', 'shack', 0] ]
df = pd.DataFrame(data, columns = ['factorA', 'factorB', 'count'])
df

but what I want is

import pandas as pd
data = [['dog', 'house', 1], ['dog', 'apartment', 2], ['dog', 'trailer', 1], ['dog', 'cabin', 0], ['dog', 'shack', 1],
['cat', 'house', 3], ['cat', 'apartment', 1], ['cat', 'trailer', 0], ['cat', 'cabin', 0], ['cat', 'shack', 3],
['gecko', 'house', 0], ['gecko', 'apartment', 3], ['gecko', 'trailer', 2], ['gecko', 'cabin', 0], ['gecko', 'shack', 0]]
df = pd.DataFrame(data, columns = ['factorA', 'factorB', 'count'])
df

Can anyone help me figure out how to do this for an arbitrary dataset which may include more than two factors?

Answer 1

Use DataFrame.set_index with DataFrame.reindex and MultiIndex.from_product :

df = df.set_index(['factorA','factorB'])
df = df.reindex(pd.MultiIndex.from_product(df.index.levels), fill_value=0).reset_index()
print (df)
   factorA    factorB  count
0      cat  apartment      1
1      cat      cabin      0
2      cat      house      3
3      cat      shack      3
4      cat    trailer      0
5      dog  apartment      2
6      dog      cabin      0
7      dog      house      1
8      dog      shack      1
9      dog    trailer      1
10   gecko  apartment      3
11   gecko      cabin      0
12   gecko      house      0
13   gecko      shack      0
14   gecko    trailer      2

Or with Series.unstack for add 0 with DataFrame.stack :

df = (df.set_index(['factorA','factorB'])['count']
         .unstack(fill_value=0)
         .stack()
         .reset_index(name='count'))

Answer 2

One way could be to use df.pivot :

In [1862]: res = df.pivot('factorA', 'factorB').fillna(0).astype(int).stack().reset_index()

In [1863]: res
Out[1863]: 
   factorA    factorB  count
0      cat  apartment      1
1      cat      cabin      0
2      cat      house      3
3      cat      shack      3
4      cat    trailer      0
5      dog  apartment      2
6      dog      cabin      0
7      dog      house      1
8      dog      shack      1
9      dog    trailer      1
10   gecko  apartment      3
11   gecko      cabin      0
12   gecko      house      0
13   gecko      shack      0
14   gecko    trailer      2