蓝天统计 - 字符编码问题

Question

I am loading a data set, characters of which was encoded in ISO 8859-9 ("Latin 5") using Windows 10 OS (Microsoft has assigned code page 28599 aka Windows-28599 to ISO-8859-9 in Windows). The data set is originally in Excel.

Whenever I run an analysis, or any operation with a variable name containing a character specific to this code page (ISO 8859-9), I get an error like:

Error: undefined columns selected
BSkyFreqResults <- BSkyFrequency(vars = c("MesleÄŸi"), data = Turnudep_raw_data_5)
Error: object 'BSkyFreqResults' not found
BSkyFormat(BSkyFreqResults)

The characters ÄŸ within "MesleÄŸi" are originally one character in Turkish (g with an inverted hat on) ğ

Those variable names that contain only letters from US code page work normally in BlueSky operations.

If I try to use save as in Excel and use web option UTF-8, to convert the data to UTF-8, this does not work either. If I export it to csv file, it does not work as is, or saved as UTF-8.

How can I load this data into BlueSky so that it works?

This same data set works in Rstudio:

> Sys.getlocale('LC_CTYPE')
[1] "Turkish_Turkey.1254"

And also in SPSS: Language is set to Unicode Picture of Language settings in SPSS

It also works in Jamovi

I also get an error when I start BlueSky, that may be relevant to this problem:

Python-CFFI error

From cffi callback <function _consolewrite_ex at 0x000002A36B441F78>:
Traceback (most recent call last):
  File "rpy2\rinterface_lib\callbacks.py", line 132, in _consolewrite_ex
  File "rpy2\rinterface_lib\conversion.py", line 133, in _cchar_to_str_with_maxlen
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 15: invalid start byte

Since then I re-downloaded and re-installed BlueSky, but I still get this Python-CFFI error every time I start the software.

I want to work with BlueSky and will appreciate any help in resolving this problem. Thanks in advance

Here is a link for reproducing the problem.

The zip file contains a data source of 2 cases both in Excel and BlueSky format, a BlueSky Markdown file to show how the error is produced and an RMarkdown file for redundancy (probably useless).

UPDATE: The Python error (Python-CFFI error) appears to be related to the Region settings in Windows.

If the region is USA ( Turnudep_reprex_Windows_Region_USA-Settings.jpg ), the python error does NOT appear. If the region is Turkey ( Turnudep_reprex_Windows_Region_Turkey-Settings.jpg ) the python error DOES appear.

Unfortunately, setting the region and language to USA does eliminate the python error message but not the other problem. Still all the operations with the Turkish variable names end up with an error.

This may be a problem only the BlueSky developers may solve...

Any help or suggestion will be greatly appreciated.

Answer 1

For version 10, according to user manual chapter 15.1.3 you can adjust the encoding setting. (answer has been edited for more clarity)