Pandas Dataframe NaN values replace by no values

Question

I wanted to replace the NaN value by an empty value to write it into an mysql database. I don't want to drop ( df.dropna() ) the the full row neither to replace it by 0 using df.fillna(0) . When using df.fillna('') or df.fillna('NULL') gives an error message:

(mysql.connector.errors.DatabaseError) 1265 (01000): Data truncated for column 'log_return' at row

The data in the dataframe looks like the following:

          date     price     log_return                 
0   2017-02-14     105.800   -0.006125                          
1   2017-02-13     106.450   0.004236 
2   2017-02-10     106.000   NaN        

What I want is the following:

          date     price     log_return                 
0   2017-02-14     105.800   -0.006125                          
1   2017-02-13     106.450   0.004236 
2   2017-02-10     106.000   

Answer 1

try this:

df.where(pd.notnull(df), None)

example

df = pd.DataFrame(np.eye(3))
df = df.where(lambda x: x==1, np.nan)
df = df.where(pd.notnull(df), None)

Note that pd.fillna(None) will not work, it leaves the NaN values untouched.

source https://github.com/pandas-dev/pandas/issues/1972