R-比较两个语料库以创建一个新的语料库，该语料库具有来自语料库＃1的较高频率的单词

Question

I have two corpuses that contain similar words. similar enough that using setdiff doesn't really help my cause. So I've turned towards finding a way to extract a list or corpus (to eventually make a wordcloud) of words that are more frequent (assuming something like this would have a threshold - so maybe like 50% more frequent?) in corpus #1, compared to corpus #2.

This is everything I have right now:

> install.packages("tm")
> install.packages("SnowballC")
> install.packages("wordcloud")
> install.packages("RColorBrewer")
> library(tm)
> library(SnowballC)
> library(wordcloud)
> library(RColorBrewer)

> UKDraft = read.csv("UKDraftScouting.csv", stringsAsFactors=FALSE)
> corpus = Corpus(VectorSource(UKDraft$Report))
> corpus = tm_map(corpus, tolower)
> corpus = tm_map(corpus, PlainTextDocument)
> corpus = tm_map(corpus, removePunctuation)
> corpus = tm_map(corpus, removeWords, c("strengths", "weaknesses", "notes",  "kentucky", "wildcats", stopwords("english")))
> frequencies = DocumentTermMatrix(corpus)
> allReports = as.data.frame(as.matrix(frequencies))

> SECDraft = read.csv("SECMinusUKDraftScouting.csv", stringsAsFactors=FALSE)
> SECcorpus = Corpus(VectorSource(SECDraft$Report))
> SECcorpus = tm_map(SECcorpus, tolower)
> SECcorpus = tm_map(SECcorpus, PlainTextDocument)
> SECcorpus = tm_map(SECcorpus, removePunctuation)
> SECcorpus = tm_map(SECcorpus, removeWords, c("strengths", "weaknesses", "notes", stopwords("english")))
> SECfrequencies = DocumentTermMatrix(SECcorpus)
> SECallReports = as.data.frame(as.matrix(SECfrequencies))

So if the word "wingspan" has a 100 count frequency in corpus#2 ('SECcorpus') but 150 count frequency in corpus#1 ('corpus'), we would want that word in our resulting corpus/list.

Answer 1

I can suggest a method that might be more straightforward, based on the new text analysis package I developed with Paul Nulty. It's called quanteda, available on CRAN and GitHub .

I don't have access to your texts, but this will work in a similar fashion for your examples. You create a corpus of your two sets of documents, then add a document variable (using docvars ), and then create a document feature matrix grouping on the new document partition variable. The rest of the operations are straightforward, see the code below. Note that by default, dfm objects are sparse Matrixes, but subsetting on features is not yet implemented (next release!).

install.packages(quanteda)
library(quanteda)

# built-in character vector of 57 inaugural addreses
str(inaugTexts)

# create a corpus, with a partition variable to represent
# the two sets of texts you want to compare
inaugCorp <- corpus(inaugTexts, 
                    docvars = data.frame(docset = c(rep(1, 29), rep(2, 28))),
                    notes = "Example made for stackoverflow")
# summarize the corpus
summary(inaugCorp, 5)

# toLower, removePunct are on by default
inaugDfm <- dfm(inaugCorp, 
                groups = "docset", # by docset instead of document
                ignoredFeatures = c("strengths", "weaknesses", "notes", stopwords("english"))),
                matrixType = "dense")

# now compare frequencies and trim based on ratio threshold
ratioThreshold <- 1.5
featureRatio <- inaugDfm[2, ] / inaugDfm[1, ]
# to select where set 2 feature frequency is 1.5x set 1 feature frequency
inaugDfmReduced <- inaugDfm[2, featureRatio >= ratioThreshold]

# plot the wordcloud
plot(inaugDfmReduced)

I would recommend you pass through some options to wordcloud() (what plot.dfm() uses), perhaps to restrict the minimum number of features to be plotted.

Very happy to assist with any queries you might have on using the quanteda package.

New

Here's a stab directly at your problem. I don't have your files so cannot verify that it works. Also if your R skills are limited, you might find this challenging to understand; ditto if you have not looked at any of the (sadly limited for now) documentation for quanteda .

I think what you need (based on your comment/query) is the following:

# read in each corpus separately, directly into quanteda
mycorpus1 <- corpus(textfile("UKDraftScouting.csv", textField = "Report"))
mycorpus2 <- corpus(textfile("SECMinusUKDraftScouting.csv", textField = "Report"))
# assign docset variables to each corpus as appropriate 
docvars(mycorpus1, "docset") <- 1 
docvars(mycorpus2, "docset") <- 2
myCombinedCorpus <- mycorpus1 + mycorpus2

then proceed with the dfm step as above, substituting myCombinedCorpus for inaugTexts .

Answer 2

I am updating the answer by @ken Benoit, as it was several years old and quanteda package has gone through some major changes in syntax.

The current version should be (April 2017):

str(inaugTexts)

# create a corpus, with a partition variable to represent
# the two sets of texts you want to compare
inaugCorp <- corpus(inaugTexts, 
                docvars = data.frame(docset = c(rep(1, 29), rep(2, 29))),
                notes = "Example made for stackoverflow")
# summarize the corpus
summary(inaugCorp, 5)


inaugDfm <- dfm(comment_corpus, 
            groups = "docset", # by docset instead of document
            remove = c("<p>", "http://", "www", stopwords("english")),
            remove_punct = TRUE,
            remove_numbers = TRUE,
            stem = TRUE)