R text mining - remove special characters and quotes

Question

I'm doing a text mining task in R.

Tasks:

1) count sentences

2) identify and save quotes in a vector

Problems :

False full stops like "..." and periods in titles like "Mr."have to be dealt with.

There's definitely quotes in the text body data, and there'll be "..." in them. I was thinking to extract those quotes from the main body and save them in a vector. (there's some manipulation to be done with them too.)

IMPORTANT TO NOTE : My text data is in a Word document. I use readtext("path to .docx file") to load in R. When I view the text, quotes are just " but not \\" contrarily to the reproducible text.

path <- "C:/Users/.../"
a <- readtext(paste(path, "Text.docx", sep = ""))
title <- a$doc_id
text <- a$text

reproducible text

text <- "Mr. and Mrs. Keyboard have two children. Keyboard Jr. and Miss. Keyboard. ... 
However, Miss. Keyboard likes being called Miss. K [Miss. Keyboard is a bit of a princess ...]
 \"Mom how are you o.k. with being called Mrs. Keyboard? I'll never get it...\". "


#  splitting by "." 
unlist(strsplit(text, "\\."))

The problem is it's splitting by false full-stops Solution I tried:

# getting rid of . in titles 
vec <- c("Mr.", "Mrs.", "Ms.", "Miss.", "Dr.", "Jr.")
vec.rep <- c("Mr", "Mrs", "Ms", "Miss", "Dr", "Jr")

library(gsubfn)
# replacing . in titles
gsubfn("\\S+", setNames(as.list(vec.rep), vec), text)

The problem with this is that it's not replacing [Miss. by [Miss

To identify quotes :

stri_extract_all_regex(text, '"\\S+"')

but that's not working too. (It's working with \\" with the code below)

stri_extract_all_regex("some text \"quote\" some other text", '"\\S+"')

The exact expected vector is :

sentences <- c("Mr and Mrs Keyboard have two children. ", "Keyboard Jr and Miss Keyboard.", "However, Miss Keyboard likes being called Miss K [Miss Keyboard is a bit of a princess ...]", ""Mom how are you ok with being called Mrs Keyboard? I'll never get it...""

I wanted the sentences separated (so I can count how many sentences in each paragraph). And quotes also separated.

quotes <- ""Mom how are you ok with being called Mrs Keyboard? I'll never get it...""

Answer 1

You may match all your current vec values using

gsubfn("\\w+\\.", setNames(as.list(vec.rep), vec), text)

That is, \\w+ matches 1 or more word chars and \\. matches a dot.

Next, if you just want to extract quotes, use

regmatches(text, gregexpr('"[^"]*"', text))

The " matches a " and [^"]* matches 0 or more chars other than " .

If you plan to match your sentences together with quotes, you might consider

regmatches(text, gregexpr('\\s*"[^"]*"|[^"?!.]+[[:space:]?!.]+[^"[:alnum:]]*', trimws(text)))

Details

• \\\\s* - 0+ whitespaces
• "[^"]*" - a " , 0+ chars other than " and a "
• | - or
• [^"?!.]+ - 0+ chars other than ? , " , ! and .
• [[:space:]?!.]+ - 1 or more whitespace, ? , ! or . chars
• [^"[:alnum:]]* - 0+ non-alphanumeric and " chars

R sample code:

> vec <- c("Mr.", "Mrs.", "Ms.", "Miss.", "Dr.", "Jr.")
> vec.rep <- c("Mr", "Mrs", "Ms", "Miss", "Dr", "Jr")
> library(gsubfn)
> text <- gsubfn("\\w+\\.", setNames(as.list(vec.rep), vec), text)
> regmatches(text, gregexpr('\\s*"[^"]*"|[^"?!.]+[[:space:]?!.]+[^"[:alnum:]]*', trimws(text)))
[[1]]
[1] "Mr and Mrs Keyboard have two children. "                                                       
[2] "Keyboard Jr and Miss Keyboard. ... \n"                                                         
[3] "However, Miss Keyboard likes being called Miss K [Miss Keyboard is a bit of a princess ...]\n "
[4] "\"Mom how are you o.k. with being called Mrs Keyboard? I'll never get it...\""