如何根据语料库的相关性制作一袋单词

Question

I understand that TF-IDF(term frequency-inverse document frequency) is the solution here? But see, TF of the TF-IDF is specific to a single document only. I need to produce a bag of words that are relevant to the WHOLE corpus. Am I doing this wrong or is there an alternative?

Answer 1

You may be able to do this if you count the IDF on a different corpus. A general corpus containing newswire texts may be suitable. Then you can treat your own corpus as a single document to count the TF. You will also need a strategy for the words that are present in your corpus but not present in the external corpus as they won't have a IDF value. Finally, you can rank the words in your corpus according to the TF-IDF.