使用来自 PCA 的组件进行分类

Question

I used PCA analysis on my dataset like so:

from sklearn.decomposition import PCA
pca = PCA(n_components=3)
principalComponents = pca.fit_transform(scale_x)
principalDf = pd.DataFrame(data=principalComponents, columns = ['PC1', 'PC2', 'PC3'])

and then on visualizing the results with MatPlotLib - I can see a division between my two classes like so:

from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(principalDf['PC1'].values, principalDf['PC2'].values, principalDf['PC3'].values, c=['red' if m==0 else 'green' for m in y], marker='o')

ax.set_xlabel('PC1')
ax.set_ylabel('PC2')
ax.set_zlabel('PC3')

plt.show()

but then when I use a classification model like SVM or Logistic Regression, it is unable to learn this relation:

from sklearn.linear_model import LogisticRegression
lg = LogisticRegression(solver = 'lbfgs')
lg.fit(principalDf.values, y)
lg_p = lg.predict(principalDf.values)
print(classification_report(y, lg_p, target_names=['Failure', 'Success']))

 precision recall f1-score support Failure 1.00 0.03 0.06 67 Success 0.77 1.00 0.87 219 accuracy 0.77 286 macro avg 0.89 0.51 0.46 286 weighted avg 0.82 0.77 0.68 286

What could be the reason for this?

Answer 1

First, use three features PC1, PC2, PC3. Additional features (PC4 ~ PC6), which is not expressed in the graph, may affects the classification result.

Second, a classifier sometimes is not trained well as you think. I recommend to use decision tree instead of the classifiers you use, because tree is (horizon) linear classifier and it would be yield the result you think.

Answer 2

Regardless of whether your results are making sense or not, you're doing something fundamentally wrong here, which is to train the classifier on the entire dataset and testing results on seen data . I've reproduced your problem using the iris dataset, and fitting a logistic regressor, has yielded good results for me:

from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split

data = load_iris()
X = data.data
y = data.target

pca = PCA(n_components=3)
principalComponents = pca.fit_transform(X)
principalDf = pd.DataFrame(data=principalComponents, columns = ['PC1', 'PC2', 'PC3'])

from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(principalDf['PC1'].values, 
           principalDf['PC2'].values, 
           principalDf['PC3'].values, 
           c=[['red', 'green', 'blue'][m] for m in y], marker='o')

ax.set_xlabel('PC1')
ax.set_ylabel('PC2')
ax.set_zlabel('PC3')

plt.show()

Now if we try to predict on X_test , we see that the confusion matrix is looking quite good in this case, meaning the the overall idea should work well:

X_train, X_test, y_train, y_test = train_test_split(principalDf, y)

lg = LogisticRegression(solver = 'lbfgs')
lg.fit(X_train, y_train)
y_pred = lg.predict(X_test)

confusion_matrix(y_true=y_test, y_pred=y_pred)

array([[ 9,  0,  0],
       [ 0, 12,  1],
       [ 0,  0, 16]], dtype=int64)