sklearn how to deal with missing_values='?'

Question

Suppose I have a .txt file, inside which is

2,3,4,?,5

I want to replace the missing value '?' by the mean of all the other data, any good idea? What if string list, I want to replace '?' by the most frequent string, ie by 'a'

'a','b','c','?','a','a'

I tried some methods, they won't work. I firstly use

import numpy as np
from sklearn.preprocessing import Imputer

row = np.genfromtxt('a.txt',missing_values='?',dtype=float,delimiter=',',usemask=True)
# this will give: row = [2 3 4 -- 5]. I checked it will use filling_values=-1 to replace missing data
# but if I add 'filling_values=np.nan' in it, it will cause error,'cannot convert float into int'

imp = Imputer(missing_values=-1, strategy='mean')
imp.fit_transform(row)
# this will give: array([2., 3., 4.,5.], which did not replace missing_value by mean value.

If I can replace '?' with np.nan, I think I could make it.

Answer 1

I couldn't reproduce the error you described, 'cannot convert float into int'.

try this:

>>> row = np.genfromtxt('a.txt',missing_values='?',dtype=float,delimiter=',')
>>> np.mean(row[~np.isnan(row)])
3.5
>>> mean = np.mean(row[~np.isnan(row)])
>>> row[np.isnan(row)] = mean
>>> row
array([ 2. ,  3. ,  4. ,  3.5,  5. ])

Edit

If you're hoping to use strings, here is a solution using ordinary lists.

>>> row = ['a','b','c','?','c','b','?','?','b']
>>> from collections import Counter
>>> letter_counts = Counter(letter for letter in row if letter != '?')
>>> letter_counts.most_common(1)
[('b', 3)]
>>> most_common_letter = letter_counts.most_common(1)[0][0]
>>> [letter if letter != '?' else most_common_letter
...     for letter in row]
['a', 'b', 'c', 'b', 'c', 'b', 'b', 'b', 'b']