子集数据帧扩展/切片以匹配原始数据帧列

Question

The Problem:

I trained a classifier on a dataset with more features than the test data. For example, my original dataset has 7 days of the week: Monday-Sunday, where-as in the test dataset, every single observation happens to be on Thursday (thus I have 6 less features). Hence, when I run the predict() , I get an error that the number of features do not match. These missing features are the features that were not created using get_dummies() :

Day_of_the_week_is_monday, Day_of_the_week_is_tuesday, ... 

Ideally, I would like to perform data cleaning and do the following:

• Automatically create missing columns, filled with 0s. (Since is_Thursday will have all 1s, the rest should have 0s)
• Remove any 'extra' columns in the subset dataframe, that are not present in the original training data. For example, get_dummies() might create more levels in the subset dataframe, which I would like to remove.

---

Reproducible Example

---

dataframe = pd.DataFrame({
                          'Result' : np.array([1,2,2,10,100],dtype='int32'),
                          'Day_of_the_week' : pd.Categorical(["Monday","Tuesday","Wednesday","Thursday","Friday"]),})

dataframe_dummies = pd.get_dummies(dataframe, prefix=['Day_of_the_week_is'])

### get subset dataframe

dataframe_subset = pd.DataFrame({
                          'Result' : np.array([1,2,2,10],dtype='int32'),
                          'Day_of_the_week' : pd.Categorical(["Thursday","Thursday","Thursday","Saturday"]),})

dataframe_subset_dummies = pd.get_dummies(dataframe_subset, prefix=['Day_of_the_week_is'])

Main dataset looks like:

   Result  Is_Friday  Is_Monday  Is_Thursday  Is_Tuesday  Is_Wednesday
0       1          0          1            0           0             0
1       2          0          0            0           1             0
2       2          0          0            0           0             1
3      10          0          0            1           0             0
4     100          1          0            0           0             0

Subset Dataframe

   Result  Day_is_Saturday  Day_is_Thursday
0       1                0                1
1       2                0                1
2       2                0                1
3      10                1                0

What has to be done:

1) Remove the is_Saturday because it's not in the original data.

2) Add remaining cols filled with 0s.

I can do it manually, but it seems very troublesome to do. Is there a function that can do this for me? Eg extend the subset dataframe to match the main data set, or remove cols to match main data?

Answer 1

A simple loop and check should do the trick to add missing columns and delete missing columns:

In [16]: a = pd.DataFrame([[1,2,3],[2,3,4]], columns=['A', 'B', 'E'])

In [17]: b = pd.DataFrame([[3,4,5],[4,5,6]], columns=['A', 'B', 'C'])

In [18]: for col in b.columns:
    ...:     if col not in a:
    ...:         a[col] = 0
    ...:

In [19]: for col in a.columns:
    ...:     if col not in b:
    ...:         del a[col]
    ...:

In [20]: a
Out[20]:
   A  B  C
0  1  2  0
1  2  3  0