R-获取documenttermmatrix中每个文档的令牌计数

Question

The reason I want to do this is so I can convert absolute frequencies into relative frequencies. It is easy to get the token count for every document but I'm not sure how to get the total token count for every document and use it at the same time so I can do /total token count for every document at the same time, is there any way to bind the rowsums and then use the column in the calculstion, if this is the correct way to do this?

Thanks

Answer 1

Using blog data from the English version of the heliohost corpus as my text data, it's pretty easy to get token counts by document via the quanteda package.

library(readr)
library(quanteda)
blogFile <- "./capstone/data/en_US.blogs.txt"
inFile <- blogFile
blogData <- read_lines(blogFile)

system.time(theText <- corpus(blogData))

head(summary(theText))

...and the output is:

> head(summary(theText))
Corpus consisting of 899288 documents, showing 100 documents:

  Text Types Tokens Sentences
 text1    18     20         1
 text2     6      7         1
 text3   104    154         7
 text4    36     43         1
 text5    91    119         5
 text6    13     13         1

Source:  C:/Users/leona/gitrepos/datascience/* on x86-64 by leona
Created: Sat Dec 02 20:59:23 2017
Notes:    
>

Answer 2

Thank you. In fact, I think I found a method, to divide by the rowSums(dtm). I hope this is the correct approach.