使用readtext进行编码

Question

I want to do some text analysis based on data stored as a .csv file, but I run into problems regarding the encoding with the readtext package.

To illustrate my problem, I created the following file in Excel, saving it as .csv (UTF-8):

|---------------------|------------------|
|      c_text         |       c_id       |
|---------------------|------------------|
|      München        |        aa        |
|---------------------|------------------|
|       Laïrie        |        bb        |
|---------------------|------------------|
|        Mános        |        cc        |
|---------------------|------------------|

Then, I load the data in R as follows:

text_raw <- readtext::readtext("path/test_encoding.csv"),
                   encoding = "UTF-8",
                   text_field = "c_text")
text_raw

The output is:

readtext object consisting of 3 documents and 1 docvar.
# Description: data.frame [3 x 3]
  doc_id              text              c_id 
  <chr>               <chr>             <chr>
1 test_encoding.csv.1 "\"München\"..." aa   
2 test_encoding.csv.2 "\"Laïrie\"..."  bb   
3 test_encoding.csv.3 "\"Mános\"..."   cc 

If I then write the object to a .csv file, the output is once again different. The command write.csv(text_raw, file = "path", fileEncoding = "UTF-8") yields the following:

München
Laïrie
Mános

Some additional information:

• I am using a Windows machine, and my sys.getLocale() is English_United Kingdom.1252 (apparently, this cannot be changed to UTF-8)

• Even if I specify other encodings in the readtext() function, (eg, "utf8", "Windows-1252", "ISO8859-1"), the output doesn't change. However, given that I explicitly save the test file as utf-8, I don't understand what's going on.

Any help would be greatly appreciated. Thanks.

Answer 1

I created a pull request since this was an issue in the readtext package:

https://github.com/quanteda/readtext/pull/151

Until this PR is either accepted or the problem otherwise fixed, you can use my fork to solve this problem:

remotes::install_github("JBGruber/readtext")

Update

The PR was approved so install the new package version via:

remotes::install_github("quanteda/readtext")

And then it should work:

df <- structure(list(c_text = structure(c(3L, 1L, 2L), .Label = c("Laïrie", 
                                                                  "Mános", "München"), class = "factor"), c_id = structure(1:3, .Label = c("aa", 
                                                                                                                                           "bb", "cc"), class = "factor")), class = "data.frame", row.names = c(NA, 
                                                                                                                                                                                                                -3L))
write.csv(df,
          "~/test.csv",
          row.names = FALSE,
          fileEncoding = "UTF-8")
text_raw <- readtext::readtext("~/test.csv",
                               encoding = "UTF-8",
                               text_field = "c_text")

text_raw
#> readtext object consisting of 3 documents and 1 docvar.
#> # Description: data.frame [3 x 3]
#>   doc_id     text             c_id 
#>   <chr>      <chr>            <chr>
#> 1 test.csv.1 "\"München\"..." aa   
#> 2 test.csv.2 "\"Laïrie\"..."  bb   
#> 3 test.csv.3 "\"Mános\"..."   cc

^{Created on 2019-05-02 by the reprex package (v0.2.1)}