Sklearn Logistic回归形状错误，但x，y形状一致

Question

I get a ValueError: Found input variables with inconsistent numbers of samples: [20000, 1] when I run the following even though the row values of x and y are correct. I load in the RCV1 dataset, get indices of the categories with the top x documents, create list of tuples with equal number of randomly-selected positives and negatives for each category, and then finally attempt to run a logistic regression on one of the categories.

import sklearn.datasets
from sklearn import model_selection, preprocessing
from sklearn.linear_model import LogisticRegression
from matplotlib import pyplot as plt
from scipy import sparse

rcv1 = sklearn.datasets.fetch_rcv1()

def get_top_cat_indices(target_matrix, num_cats):
    cat_counts = target_matrix.sum(axis=0)
    #cat_counts = cat_counts.reshape((1,103)).tolist()[0]
    cat_counts = cat_counts.reshape((103,))

    #b = sorted(cat_counts, reverse=True)
    ind_temp = np.argsort(cat_counts)[::-1].tolist()[0]

    ind = [ind_temp[i] for i in range(5)]
    return ind

def prepare_data(x, y, top_cat_indices, sample_size):
    res_lst = []

    for i in top_cat_indices:

        # get column of indices with relevant cat
        temp = y.tocsc()[:, i]

        # all docs with labeled category
        cat_present = x.tocsr()[np.where(temp.sum(axis=1)>0)[0],:]
        # all docs other than labelled category
        cat_notpresent = x.tocsr()[np.where(temp.sum(axis=1)==0)[0],:]
        # get indices equal to 1/2 of sample size
        idx_cat = np.random.randint(cat_present.shape[0], size=int(sample_size/2))
        idx_nocat = np.random.randint(cat_notpresent.shape[0], size=int(sample_size/2))
        # concatenate the ids

        sampled_x_pos = cat_present.tocsr()[idx_cat,:]
        sampled_x_neg = cat_notpresent.tocsr()[idx_nocat,:]
        sampled_x = sparse.vstack((sampled_x_pos, sampled_x_neg))

        sampled_y_pos = temp.tocsr()[idx_cat,:]
        sampled_y_neg = temp.tocsr()[idx_nocat,:]
        sampled_y = sparse.vstack((sampled_y_pos, sampled_y_neg))

        res_lst.append((sampled_x, sampled_y))

    return res_lst

ind = get_top_cat_indices(rcv1.target, 5)
test_res = prepare_data(train_x, train_y, ind, 20000)

x, y = test_res[0]
print(x.shape)
print(y.shape)
LogisticRegression().fit(x, y)

Could it be an issue with the sparse matrices, or problem with dimensionality (there are 20K samples and 47K features)

Answer 1

When I run your code, I get following error:

AttributeError: 'bool' object has no attribute 'any'

That's because y for LogisticRegression needs to numpy array. So, I changed last line to:

LogisticRegression().fit(x, y.A.flatten())

Then I get following error:

ValueError: This solver needs samples of at least 2 classes in the data, but the data contains only one class: 0

This is because your sampling code has a bug . You need to subset y array with rows having that category before using sampling indices. See code below:

def prepare_data(x, y, top_cat_indices, sample_size):
    res_lst = []

    for i in top_cat_indices:

        # get column of indices with relevant cat
        temp = y.tocsc()[:, i]

        # all docs with labeled category
        c1 = np.where(temp.sum(axis=1)>0)[0]
        c2 = np.where(temp.sum(axis=1)==0)[0]
        cat_present = x.tocsr()[c1,:]
        # all docs other than labelled category
        cat_notpresent = x.tocsr()[c2,:]
        # get indices equal to 1/2 of sample size
        idx_cat = np.random.randint(cat_present.shape[0], size=int(sample_size/2))
        idx_nocat = np.random.randint(cat_notpresent.shape[0], size=int(sample_size/2))
        # concatenate the ids

        sampled_x_pos = cat_present.tocsr()[idx_cat,:]
        sampled_x_neg = cat_notpresent.tocsr()[idx_nocat,:]
        sampled_x = sparse.vstack((sampled_x_pos, sampled_x_neg))

        sampled_y_pos = temp.tocsr()[c1][idx_cat,:]
        print(sampled_y_pos.nnz)
        sampled_y_neg = temp.tocsr()[c2][idx_nocat,:]
        print(sampled_y_neg.nnz)

        sampled_y = sparse.vstack((sampled_y_pos, sampled_y_neg))

        res_lst.append((sampled_x, sampled_y))

    return res_lst

Now, Everything works like a charm