在命令行中有条件地在perl中将默认编码设置为utf-8

Question

In order to process text in utf-8 in Perl, I have been using binmode(<file-handle>, ":encoding(UTF-8)"); on each stream I use. I just discovered that

use open ( ":encoding(UTF-8)", ":std" );

can be used to do the same thing globally. This is great, since it means a lot less repetitive code.

But now I have a problem: I would like to have a command line option to my script, -utf8 , which turns everything utf-8 only when supplied. Since use open is a pragma, it is lexically scoped and I cannot put it in an if statement, but without an if statement it cannot depend on command line options.

Here is a minimal example illustrating the problem, call it problem.pl

#!/usr/bin/env perl

# hard-coded in my minimal example, normally set by command line option -utf8
my $use_utf8 = 1;

# use only applies within its lexical scope - this does not work
if ($use_utf8) {
   use open ( ":encoding(UTF-8)", ":std" );
}

# if I put it at the right lexical scope, it's not conditional on $use_utf8
#..e open ( ":encoding(UTF-8)", ":std" );

while (<>) {
   print length($_);
}

When I run this code on a file, call in input , containing one line with a 2-byte UTF-8 character, say à , it outputs 3:

$ ./problem.pl input
3

If I move the use open statement to the global scope, I get the expected results of a length of 2 (one character plus one newline):

$ ./problem.pl input
2

So how can I set the encoding to utf-8 globally, but conditionally on a command-line option, so that I would get 2 with -utf8 but 3 without.

Also, in my real use case, I use the spaceship operator ( while (<>) ) to provide high flexibility in the command line syntax to process multiple files, but in this case I can't call binmode since the file handles are managed automatically by Perl. use open would be a much nicer option, if I could make it conditional.

PS: Yes, I really do still have non-utf8 data that I want to continue to be able to handle. Thank God most of our data is now in utf-8, but unfortunately not all of it yet.

Answer 1

First: you can use if to conditionally apply a lexical pragma. Just make sure the condition is available at compile time (you may need to use a BEGIN block before).

my $use_utf8;
BEGIN { $use_utf8 = 1; }
use if $use_utf8, 'open', ':std', ':encoding(UTF-8)';

The -C option works similarly to the open pragma for utf8 layers. -CSD will set it on the standard handles (S) and any handles opened (D). Unfortunately it uses the less safe :utf8 layer instead of :encoding(UTF-8) , so you may end up with broken strings if you use it for input that is not actually UTF-8. Also, -CD sets a default for any handles opened in the whole program, not just the lexical scope of your script, this can possibly break usage of modules that don't expect it. ( -CS is always global, as is the ':std' effect of the open pragma, since the standard handles are global.)

perl -CSD problem.pl input