如何在两列上使用 sklearn TfidfVectorizer fit_transform

Question

Not sure if this the correct way to apply fit_transform on both of these columns. Currently am writing a classifier to predict fraudulent job postings. I'm interested in 'description' and 'requirements' columns. I don't know if there is a way to do both transforms in the same line.

preprocessor = TfidfVectorizer(stop_words='english', strip_accents='unicode', norm='l2', use_idf=False,smooth_idf=False)
XX = preprocessor.fit_transform(X["description"])
XX = preprocessor.fit_transform(X["requirements"])

Answer 1

I think that you are misinterpreting the documentation. If you want to do tfidf on two columns, then you need to pass two transformers. Something like this:

tfidf_1 = TfidfVectorizer(min_df=0)
tfidf_2 = TfidfVectorizer(min_df=0)
clmn = ColumnTransformer([("tfidf_1", tfidf_1, "a"), 
                          ("tfidf_2", tfidf_2, "b")
                         ],
                         remainder="passthrough")