DataFrameMapper scikit-learn ValueError: all the input array dimensions except for the concatenation axis must match exactly

Question

I have been trying to use DataFrameMapper to add multiple pre-processing transformations on my dataframe into my scikit-learn Pipeline.

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data"
names = ['Sex', 'Length', 'Diameter', 'Height', 'Whole weight', 'Schuked weight', 'Viscera weight', 'Shell weight', 'Rings']

df = pd.read_csv(url, names=names)

mapper = DataFrameMapper(
    [('Height', Normalizer()), ('Sex', LabelBinarizer())]
)

stages = []

stages += [("mapper", mapper)]

estimator = DecisionTreeClassifier()

stages += [("dtree", estimator)]

pipeline = Pipeline(stages)

labelCol = 'Rings'
target = df[labelCol]
data = df.drop(labelCol, axis=1)

train_data, test_data, train_target, expected = train_test_split(data, target, test_size=0.25, random_state=33)

model = pipeline.fit(train_data, train_target)

However, I am getting the following error:

Traceback (most recent call last):
  File "app/experimenter/sklearn/transformations.py", line 65, in <module>
    model = pipeline.fit(train_data, train_target)
  File "/Library/Python/2.7/site-packages/sklearn/pipeline.py", line 268, in fit
    Xt, fit_params = self._fit(X, y, **fit_params)
  File "/Library/Python/2.7/site-packages/sklearn/pipeline.py", line 234, in _fit
    Xt = transform.fit_transform(Xt, y, **fit_params_steps[name])
  File "/Library/Python/2.7/site-packages/sklearn/base.py", line 497, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/Library/Python/2.7/site-packages/sklearn_pandas/dataframe_mapper.py", line 225, in transform
    stacked = np.hstack(extracted)
  File "/Library/Python/2.7/site-packages/numpy/core/shape_base.py", line 288, in hstack
    return _nx.concatenate(arrs, 1)
ValueError: all the input array dimensions except for the concatenation axis must match exactly

What am I missing?

Thanks :)

Answer 1

You will have to alter the construction of the DataFrameMapper :

mapper = DataFrameMapper(
    [(['Height'], Normalizer()), ('Sex', LabelBinarizer())]
)

This is a subtle detail which can be found in the documentation of sklearn_pandas :

Map the Columns to Transformations

The difference between specifying the column selector as 'column' (as a simple string) and ['column'] (as a list with one element) is the shape of the array that is passed to the transformer. In the first case, a one dimensional array will be passed, while in the second case it will be a 2-dimensional array with one column, ie a column vector.

[...]

Be aware that some transformers expect a 1-dimensional input (the label-oriented ones) while some others, like OneHotEncoder or Imputer , expect 2-dimensional input, with the shape [n_samples, n_features] .