有没有办法组合这些 sklearn Pipelines/ColumnTransformers，这样我就不必进行多次 fit_transform() 调用？

Question

I'd like to create a Pipeline where I can call fit_transform() just one time on my train dataset (train_df) and receive a fully preprocessed dataset. I don't think I can currently do that, however, because I have to call PCA() on the output of a ColumnTransformer and then concatenate that output with the result of a separate ColumnTransformer called on train_df. Basically, I think I'm going too high up the abstraction ladder, with one too many pipelines/ct's embedded within each other. There's no way to streamline the entire preprocessing process by passing train_df to a single Pipeline or ColumnTransformer - unless I'm missing something and you have any insight? I've spent hours wracking myself around this problem and have finally faced the reality I'm just spinning my wheels. Any help or solutions would be greatly appreciated.

Thank you!

num_ct = ColumnTransformer([
                        ('non_skewed_num', non_skewed_num_pipe, non_skewed_vars),
                        ('skewed_num', skewed_num_pipe, skewed_vars)
                        ], remainder='drop')

total_num_pipe = Pipeline([('num_ct', num_ct), 
                           ('dim_reduc', PCA(n_components=5))])


cat_ct = ColumnTransformer([
                        ('cat_pipe1', cat_pipe1, cat_vars1),
                        ('cat_pipe2', cat_pipe2, cat_vars2)
                        ], remainder='drop')


final_num = total_num_pipe.fit_transform(train_df)
final_cat = cat_ct.fit_transform(train_df)
final_X_train = np.c_[final_num, final_cat]

Answer 1

I finally found a solution to this, thanks to @Alexander's suggestion of chaining ColumnTransformers into a Pipeline. (TLDR: Don't forget that you can create a Pipeline of ColumnTransformers, using remainder='passthrough' to your advantage.)

I first created a ColumnTransformer that concatenates the transformations for both numeric and categorical variables, but without the PCA.

ct = ColumnTransformer([
                        ('non_skewed_num', non_skewed_num_pipe, non_skewed_vars),
                        ('skewed_num', skewed_num_pipe, skewed_vars),
                        ('cat_pipe1', cat_pipe1, cat_vars1),
                        ('cat_pipe2', cat_pipe2, cat_vars2)
                        ], remainder='drop')

Then, I created a ColumnTransformer just for the PCA, and when I specified which columns to apply this to, I used a slice object since this ColumnTransformer will be fed a NumPy array--not a DataFrame--in the eventual Pipeline (it will be the second ColumnTransformer in the Pipeline). I also set remainder='passthrough' so the non-numeric variables will be retained untransformed after the PCA.

ct2 = ColumnTransformer([('dim_reduc', PCA(n_components=5), slice(0, 37))], remainder='passthrough')  # 37 is number of numeric variables

Finally, I created a Pipeline chaining these two ColumnTransformers

final_pipe = Pipeline([('ct', ct), 
                       ('ct2', ct2)])

Calling final_pipe.fit_transform(train_df) yields the cleaned array I wanted. Hope this helps!