R tm包和Spark / python为文档术语频率任务提供了不同的词汇量

Question

I have a csv with a single column, each row is a text document. All text has been normalized:

• all lowercase
• no punctuation
• no numbers
• no more than one whitespace between words
• no tags(xml, html)

I have also this R script which constructs the Document Term Matrix on these documents and does some machine learning analysis. I need to convert this in Spark.

The first step is to produce the Document Term Matrix where for each term there is the relative frequency count in the document. The problem is that I am getting different vocabularies size using R, respect to spark api or python sklearn (spark and python are consistent in the result).

This is the relevant code for R:

library(RJDBC)
library(Matrix)
library(tm)
library(wordcloud)
library(devtools)
library(lsa)
library(data.table)
library(dplyr)
library(lubridate)

corpus <- read.csv(paste(inputDir, "corpus.csv", sep="/"), stringsAsFactors=FALSE)
DescriptionDocuments<-c(corpus$doc_clean)
DescriptionDocuments <- VCorpus(VectorSource(DescriptionDocuments))
DescriptionDocuments.DTM <- DocumentTermMatrix(DescriptionDocuments, control = list(tolower = FALSE,
                                                                                    stopwords = FALSE,
                                                                                    removeNumbers = FALSE,
                                                                                    removePunctuation = FALSE,
                                                                                    stemming=FALSE))

# VOCABULARY SIZE = 83758

This is the relevant code in Spark (1.6.0, Scala 2.10):

import org.apache.spark.ml.feature.{CountVectorizer, CountVectorizerModel, RegexTokenizer}

var corpus = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "false").load("/path/to/corpus.csv")

// RegexTokenizer splits by default on one or more spaces, which is ok
val rTokenizer = new RegexTokenizer().setInputCol("doc").setOutputCol("words")
val words = rTokenizer.transform(corpus)

val cv = new CountVectorizer().setInputCol("words").setOutputCol("tf")
val cv_model = cv.fit(words)
var dtf = cv_model.transform(words)

// VOCABULARY SIZE = 84290

I've also checked in python sklearn and I got consistent result with spark:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

corpus = pd.read_csv("/path/to/corpus.csv")
docs = corpus.loc[:, "doc"].values

def tokenizer(text):
    return text.split

cv = CountTokenizer(tokenizer=tokenizer, stop_words=None)
dtf = cv.fit_transform(docs)
print len(dtf.vocabulary_)

# VOCABULARY SIZE = 84290

I don't know very much R tm package but it seems to me that by default should tokenize on white spaces by default. Someone has any hint why am I getting different vocabulary size?

Answer 1

The reason for the difference is a default option within the creation of a document term matrix. If you check ?termFreq you can find the option wordLengths:

An integer vector of length 2. Words shorter than the minimum word length wordLengths[1] or longer than the maximum word length wordLengths[2] are discarded. Defaults to c(3, Inf), ie, a minimum word length of 3 characters.

The default setting of c(3, Inf) removes all words shorter than 3, like "at", "in", "I", etc etc.

This default is what is causing the difference between tm and spark / python

See the difference in the wordLengths setting in the example below.

library(tm)

data("crude")

dtm <- DocumentTermMatrix(crude)
nTerms(dtm)
[1] 1266

dtm2 <- DocumentTermMatrix(crude, control = list(wordLengths = c(1, Inf)))
nTerms(dtm2)
[1] 1305