如何聚类来自不同文件的不同文本？

Question

I would like to cluster texts from different files to their topics. I am using the 20 newsgroup dataset. So there are different categories and I would like to cluster the texts to these categories with DBSCAN. My problem is how to do this?

At the moment I am saving each text of a file in a dict as a string. Then, I am removing several characters and words and extracting nouns from each dict entry. Then, I would like to apply Tf-idf on each dict entry which works but how can I pass this to DBSCAN to cluster this in categories?

my text processing and data handling:

counter = 0
dic = {}
for i in range(len(categories)):
            path = Path('dataset/20news/%s/' % categories[i])
            print("Getting files from: %s" %path)
            files = os.listdir(path)
            for f in files:
                with open(path/f, 'r',encoding = "latin1") as file:
                    data = file.read()
                    dic[counter] = data
                    counter += 1

if preprocess == True:
        print("processing Data...")
        content = preprocessText(data)
if get_nouns == True:
        content = nounExtractor(content)
tfidf_vectorizer = TfidfVectorizer(stop_words=stop_words, max_features=max_features)
for i in range(len(content)):
        content[i] = tfidf_vectorizer.fit_transform(content[i])

So I would like to pass each text to DBSCAN and I think it would be wrong to put all texts in one string because then there is no way to assign labels to it, am I right?

I hope my explanation is not too confusing.

Best regards!

EDIT:

 for f in files:
                with open(path/f, 'r',encoding = "latin1") as file:
                    data = file.read()
                    all_text.append(data)

tfidf_vectorizer = TfidfVectorizer(stop_words=stop_words, max_features=max_features)
    tfidf_vectorizer.fit(all_text)
    text_vectors = [] 
    for text in all_text: 
        text_vectors.append(tfidf_vectorizer.transform(text))

Answer 1

You should fit the TFIDF vectorizer to the whole training text corpus, and then create a vector representation for each text/document on it's own by transforming it using the TFIDF, you should then apply clustering to those vector representation for the documents.

EDIT

A simply edit to your original code would be instead of the following loop

for i in range(len(content)):
        content[i] = tfidf_vectorizer.fit_transform(content[i])

You could do this

transformed_contents = tfidf_vectorizer.fit_transform(content)

transformed_contents will then contain the vectors that you should run your clustering algorithm against.