即使要求使用 UTF-8 对其进行编码，Perl output 仍无法读取

Question

As is apparent in the title of my question I use Perl to do some regex related query. However, I use UTF8 encoded input with diacritics in the body and when I generate an output it is always UTF-16 LE / UCS-2 LE BOM encoded.

My main problem is that the diacritics in the input are replaced with '??' in the output. I think this problem lies in the encoding. I have tried to strip a lot of the code which could have been responsible for my problem, but the problem persisted.

This is my code:

# Input = élèvàtòr ôpëràtör
# Output = ??l??v??t??r ??p??r??t??r

use utf8;
use open qw(:std :utf8);

    {
    while (<STDIN>)
        {
        $line = $_;

        # remove long span (), <>
        $line =~ s/[\(\)]//g;

        # remove long span [] with everything in between
        while ($line =~ s/\[[^\[\]]*\]//g) {;}
        while ($line =~ s/\<[^\<\>]*\>//g) {;}

        printf("$line");
        }
    }

I have changed it multiple times as to the suggestions of several other answers to similar questions here on this site:

I have tried this option:

use utf8;  # Source is encoded using UTF-8
use open ':std', ':encoding(locale)';

This resulted in my CLI stating

Cannot find encoding "locale" at /usr/share/perl5/core_perl/open.pm line 126.
Cannot find encoding "locale" at /usr/share/perl5/core_perl/open.pm line 134.

I have also tried running the following options tagged at the end of my command in my CLI:

-CDSL -le 'print "\x{1815}"'
-CO

More lines I put in my code without success:

binmode(STDOUT, ":utf8");
use open ":encoding(utf8)";
use open IN => ":encoding(utf8)", OUT => ":utf8";

Someone also recommended to use the

'environment'

but I couldn't find anything as to how I would do that. The code he suggested was:

export PERL_UNICODE=SDL

But I don't know where to put this or alter it.

I hope someone can help me with this problem.

Answer 1

The output you showed does not appear to be UTF-16 or UCS-2 as you claim. (There are too few ? .) There's also no evidence of a BOM.

On the other hand, the output is consistent with UTF-8. é , è , à , ò , ô , ë , à and ö all encode as two bytes using UTF-8.

Perl is doing exactly what you asked, but you are viewing UTF-8 with a tool or terminal that expects a different encoding. You need to provide the proper encoding for your tool or terminal, or adjust what your tool or terminal expects.

For example, you can tell a Windows Console to expect UTF-8 by using chcp 65001 .

Since you provided absolutely no information about your tool or terminal, this is as far as we can help.