How to get 10 individual confusion matrices using 10-fold cross validation using sklearn

Question

I'm new to machine leaning so this is my first time using sklearn packages. In this classification problem I want to get confusion matrix for each fold, but I get only one, this is what I have done so far. I haven't added the preprocessing part here.

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import cross_val_predict

target = df["class"]
features = df.drop("class", axis=1)
split_df = round(0.8 * len(df))

features = features.sample(frac=1, random_state=0)
target = target.sample(frac=1, random_state=0)

trainFeatures, trainClassLabels = features.iloc[:split_df], target.iloc[:split_df]
testFeatures, testClassLabels = features.iloc[split_df:], target.iloc[split_df:]

tree = DecisionTreeClassifier(random_state=0)
tree.fit(X=trainFeatures, y=trainClassLabels)

y_pred = cross_val_predict(tree, X=features, y=target, cv=10)

conf_matrix = confusion_matrix(target, y_pred)
print("Confusion matrix:\n", conf_matrix)

Answer 1

You would need to provide the split using Kfold , instead of specifying cv=10. For example:

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import KFold, cross_val_predict
from sklearn.datasets import make_classification

features, target = make_classification(random_state=0)

tree = DecisionTreeClassifier(random_state=0)
kf = KFold(10,random_state=99,shuffle=True)

y_pred = cross_val_predict(tree, X=features, y=target, cv=kf)

conf_matrix = confusion_matrix(target, y_pred)
print("Confusion matrix:\n", conf_matrix)

Confusion matrix:
 [[41  9]
 [ 6 44]]

Then we can make the confusion matrix for each fold:

lst = []
for train_index, test_index in kf.split(features):
    lst.append(confusion_matrix(target[test_index], y_pred[test_index]))
    

It looks like this:

[array([[4, 0],
        [0, 6]]),
 array([[4, 3],
        [1, 2]]),
 array([[2, 0],
        [2, 6]]),
 array([[5, 1],
        [0, 4]]),
 array([[4, 1],
        [1, 4]]),
 array([[2, 2],
        [0, 6]]),
 array([[4, 0],
        [0, 6]]),
 array([[4, 1],
        [1, 4]]),
 array([[4, 1],
        [1, 4]]),
 array([[8, 0],
        [0, 2]])]