稀疏矩阵上的scikit-learn Normalizer过程

Question

I m try to normalize data in sparse matrix (matrix is in TF format). i had a doubt,

It's right use sklearn.preprocessing.Normalizer just for normalize my matrix?

Does it make sense to normalize and use the TF matrix to cluster?

My matrix is in this way:

 (0, 0) 1
 (7, 0) 1
 (13, 0)    1
 (31, 0)    4
 (97, 0)    3
 (99, 0)    1

i use this code, from the sklearn API:

 transformer = Normalizer().fit(sparse_matrix) # fit does nothing.

 Normalizer(copy=True, norm='l2')
 transformer.transform(sparse_matrix)  

Where sparse matrix, is my TF matrix.

The output is this one:

 (0, 0) 0.04822428221704121
 (0, 1) 0.04822428221704121
 (0, 2) 0.04822428221704121
 (0, 3) 0.14467284665112365
 (0, 4) 0.04822428221704121
 (0, 5) 0.04822428221704121
 (0, 6) 0.09644856443408242
 (0, 7) 0.19289712886816485

Since it's the first time I've done this, I do not want to be wrong. On these new data I want to apply clustering to see differences between normalization and TF-IDF. Excuse me if this question sounds silly, but I'm trying to learn from zero.

Answer 1

The tfidf matrix produced by sklearn is already normalized in an appropriate way.

The usual normalization is to unit l2 length, in order to make the dot product, Euclidean distance, and cosine return the same ranking. From a theoretical point of view (and you should always consider the why ), this corresponds to normalizing the document length : a document that is twice the concatenation of another document will produce the same vector.

Think a few minutes how to check that the matrix is indeed normalized this way. That is a one line expression involving dot .