Why it did convert to numpy.ndarray automaticly?

Question

I was doing the division in my database between predictors and class, so I realized I had to do LabelEncoder transformations and then OneHotEncoder, in the first database I did it like this:

label_encoder_workclass = LabelEncoder()
label_encoder_education = LabelEncoder()
label_encoder_marital = LabelEncoder()
label_encoder_occupation = LabelEncoder()
label_encoder_relationship = LabelEncoder()
label_encoder_race = LabelEncoder()
label_encoder_sex = LabelEncoder()
label_encoder_country = LabelEncoder()

X_census[:,1] = label_encoder_workclass.fit_transform(X_census[:,1])
X_census[:,3] = label_encoder_education.fit_transform(X_census[:,3])
X_census[:,5] = label_encoder_marital.fit_transform(X_census[:,5])
X_census[:,6] = label_encoder_occupation.fit_transform(X_census[:,6])
X_census[:,7] = label_encoder_relationship.fit_transform(X_census[:,7])
X_census[:,8] = label_encoder_race.fit_transform(X_census[:,8])
X_census[:,9] = label_encoder_sex.fit_transform(X_census[:,9])
X_census[:,13] = label_encoder_country.fit_transform(X_census[:, 13])

onehotenconder_census = ColumnTransformer(transformers=[('OneHot', OneHotEncoder(), [1,3,5,6,7,8,9,13])], remainder='passthrough')

X_census = onehotenconder_census.fit_transform(X_census).toarray()

And in the second database like this:

label_encoder_personHomeOwnership = LabelEncoder()
label_encoder_loanIntent = LabelEncoder()
label_encoder_loanGrade = LabelEncoder()
label_encoder_cbPersonDefaultOnFile = LabelEncoder()

X_credit[:,2] = label_encoder_personHomeOwnership.fit_transform(X_credit[:,2])
X_credit[:,4] = label_encoder_loanIntent.fit_transform(X_credit[:,4])
X_credit[:,5] = label_encoder_loanGrade.fit_transform(X_credit[:,5])
X_credit[:,9] = label_encoder_personHomeOwnership.fit_transform(X_credit[:,9])

oneHotEncoder_credit = ColumnTransformer(transformers=[('OneHot', OneHotEncoder(), [2,4,5,9])], remainder='passthrough')

X_credit = oneHotEncoder_credit.fit_transform(X_credit)

What intrigued me was why in the first one I had to use the toarray() method to convert it to an object of type numpy.ndarray and in the second one I didn't, it converted automatically.

Please someone take this question from me. Did I do something wrong?

Thank you very much in advance

Answer 1

From help page of ColumnTransformer :

sparse_thresholdfloat, default=0.3

If the output of the different transformers contains sparse matrices, these will be stacked as a sparse matrix if the overall density is lower than this value. Use sparse_threshold=0 to always return dense. When the transformed output consists of all dense data, the stacked result will be dense, and this keyword will be ignored.

In your case, the first example has more sparse entries than the second, hence it was converted to a sparse matrix. The .toarray() method converts it from sparse to dense.

If memory is not an issue, setting it to sparse_threshold=0 will ensure you get a dense matrix every time.

For example if we have columns with a lot of categories:

from sklearn.compose import ColumnTransformer
import numpy as np

np.random.seed(111)

X = np.random.randint(0,10,(100,10))
ct = ColumnTransformer(transformers=[('OneHot', OneHotEncoder(),
np.arange(10))], remainder='passthrough')

type(ct.fit_transform(X))
scipy.sparse.csr.csr_matrix

ct = ColumnTransformer(transformers=[('OneHot', OneHotEncoder(),
np.arange(10))], remainder='passthrough',sparse_threshold=0)

type(ct.fit_transform(X))
numpy.ndarray

As compared to columns with less categories:

X = np.random.randint(0,2,(100,10))
ct = ColumnTransformer(transformers=[('OneHot', OneHotEncoder(),
np.arange(10))], remainder='passthrough')

type(ct.fit_transform(X))
numpy.ndarray