如何在我的数据框中找到缺失值，处理这些缺失值的最佳方法是什么？

Question

Assume the following data frame:

"?"==missing value

how can i find "?" in this data frame by python and how can i handle this missing values by the bestest method?

col1   col2     col3    col4    col5   target
?       1         ?       1      20     0
90      1         47      0      40     1
75      ?        246      ?      15     0
60      1        315      1      60     0
78      0        224      0      50     0
48      1         ?       1       ?     1
65      1        135      0      35     1
73      0        582      0      35     1
70      0        1202     0      50     1
54      1        427      0      70     1
68      1        1021     1      35     0
55      0         ?       1      35     1

Answer 1

What do you mean by "finding"?

For example, you could get the null values like this:

df.loc[df['col1'].isnull(), 'col']

There is no "best method" to fill in missing values - it depends on the nature of your data. For example, you can fill in missing values with zeros like this:

df = df.fillna(0)

Answer 2

Assuming your data frame (more details would be helpful) is a Pandas data frame named df :

import numpy as np
df = df.replace('?',np.NaN)

This replaces all ? with numpy's NaN, which then are handled accordingly in calculation and/or plotting.

If you read the data with pd.read_csv() you can directly used the na_values option:

import pandas as pd 
df = pd.read_csv('datafile.csv', na_values = "?")

Answer 3

There are multiple ways to handle missing data. Here are some of them -

• Remove rows with missing data
• Remove rows for specific variables
• Drop variables with missing data
• Impute missing data with fixed values (like 0, -1, etc)
• Impute missing data with central tendencies (like mean, median etc)
• Interpolate row sequence or columns sequence data
• Predict missing data using some ML model
• Denoising methods

Depending on the amount, type and end task you have with the dataset, each of these methods has its own place.

You can read about this in more detail with code examples in my blog Handling missing data (like a boss!)

A library I usually recommend is called missingno . This lets you analyze your missing data visually as well as with nullity correlations (existential correlation between variables). Github repo and documentation here .

---

Detailed guide

1. Get to nans and not nans

The first step would be to analyze the missing data by changing the ? to nan values, and getting a sense of how much of it exists. You can use missingno.matrix and missingno.bar to visualize this.

#!pip install missingno
import missingno as msno 
import numpy as np


df_raw = df.replace('?', np.nan)
print(df_raw)

#    col1 col2  col3 col4 col5  target
# 0   NaN    1   NaN    1   20       0
# 1    90    1    47    0   40       1
# 2    75  NaN   246  NaN   15       0
# 3    60    1   315    1   60       0
# 4    78    0   224    0   50       0
# 5    48    1   NaN    1  NaN       1
# 6    65    1   135    0   35       1
# 7    73    0   582    0   35       1
# 8    70    0  1202    0   50       1
# 9    54    1   427    0   70       1
# 10   68    1  1021    1   35       0
# 11   55    0   NaN    1   35       1

2. Row-wise analysis

You can visually represent the missing values as white and existing values as black. As you can see below, a few of your rows have too many missing values (only 4 values available). Based on this you can make a decision to keep or chuck these rows if you want.

msno.matrix(df_raw)

3. Column-wise analysis

Total number of missing values column-wise with bar chart. Your col3 has the highest number of missing values. If you replace these values with say a mean, you have to ask yourself if the mean of the remaining values is sufficient enough to represent those values!

msno.bar(df_raw)

From hereon, you have to decide the purpose of each row/column before taking a decision to handle missing data. A quick way would be to simply bump each missing value as 0. But, each column in your case represents something different. Some are binary while others are continuous.

You may want to handle that fact independently.

If you want to replace a selected column by its mean, you can simply do -

column = 'col1' #change column name
df[column] = df[column].fillna(df[column].mean())

Modifying the above code will help you handle each column separately (works for rows as well using df.iloc[] )