在 R 中使用波斯语进行文本挖掘

Question

I'm looking to do some v. simple data mining (frequency, bigrams, trigrams) on some facebook posts in Persian that I've collected and archived in a csv. Below is the script I would use with english language csv of facebook comments to unnest all individual words into their own column.

stp_tidy <- stc2 %>%
  filter(!str_detect(Message, "^RT")) %>%
  mutate(text = str_replace_all(Message, "https://t.co/[A-Za-z\\d]+|http://[A-Za-z\\d]+|&amp;|&lt;|&gt;|RT","")) %>%
  unnest_tokens(word, text, token = "regex", pattern = reg_words) %>%
  filter(!word %in% stop_words$word,
         str_detect(word, "[a-z]"))

Does anyone know of any method for applying unnest_tokens in Persian (or Dari to be specific) script?

Answer 1

2 options. First example is using quanteda, second example is using udpipe.

Note that the printing of the tibbles with farsi is weird, aka features and values tend to be printed in the wrong columns, but the data is correctly stored inside the objects for further processing. There is a slight difference in output between the 2 options. But these tend to be negligable. Note that for reading in the data I used the readtext package. This tends to play nice with quanteda.

1 quanteda

library(quanteda)
library(readtext)
# library(stopwords)

stp_test <- readtext("stp_test.csv", encoding = "UTF-8")

stp_test$Message[stp_test$Message != ""]
stp_test$text[stp_test$text != ""]

# remove records with empty messages

stp_test <- stp_test[stp_test$Message != "", ]

stp_corp <- corpus(stp_test, 
                   docid_field = "doc_id",
                   text_field = "Message")


stp_toks <- tokens(stp_corp, remove_punct = TRUE)
stp_toks <- tokens_remove(stp_toks, stopwords::stopwords(language = "fa", source = "stopwords-iso"))


# step for creating ngrams 1-3 can be done here, after removing stopwords. 
# stp_ngrams <- tokens_ngrams(stp_toks, n = 1L:3L, concatenator = "_")

stp_dfm <- dfm(stp_toks)
textstat_frequency(stp_dfm)

# transform into tidy data.frame
library(dplyr)
library(tidyr)
quanteda_tidy_out <- convert(stp_dfm, to = "data.frame") %>% 
  pivot_longer(-document, names_to = "features")

2 udpipe

library(udpipe)
model <- udpipe_download_model(language = "persian-seraji")
ud_farsi <- udpipe_load_model(model$file_model)

# use stp_test from quanteda example.
x <- udpipe_annotate(ud_farsi, doc_id = stp_test$doc_id, stp_test$Message)
stp_df <- as.data.frame(x)


# selecting only nouns and verbs and removing stopwords 
ud_tidy_out <- stp_df %>% 
  filter(upos %in% c("NOUN", "VERB"),
         !token %in% stopwords::stopwords(language = "fa", source = "stopwords-iso")) 

Both packages have a good vignettes and support pages.