[英]How to detect latin1 and UTF-8?
I am extracting strings from an XML file, and even though it should be pure UTF-8, it is not.我正在从 XML 文件中提取字符串,尽管它应该是纯 UTF-8,但事实并非如此。 My idea was to我的想法是
#!/usr/bin/perl
use warnings;
use strict;
use Encode qw(decode encode);
use Data::Dumper;
my $x = "m\x{e6}gtig";
my $y = "m\x{c3}\x{a6}gtig";
my $a = encode('UTF-8', $x);
my $b = encode('UTF-8', $y);
print Dumper $x;
print Dumper $y;
print Dumper $a;
print Dumper $b;
if ($x eq $y) { print "1\n"; }
if ($x eq $a) { print "2\n"; }
if ($a eq $y) { print "3\n"; }
if ($a eq $b) { print "4\n"; }
if ($x eq $b) { print "5\n"; }
if ($y eq $b) { print "6\n"; }
outputs产出
$VAR1 = 'm�gtig';
$VAR1 = 'mægtig';
$VAR1 = 'mægtig';
$VAR1 = 'mægtig';
3
under the idea that only a latin1 string would increase its length, but encoding an already UTF-8 also makes it longer.在只有 latin1 字符串会增加其长度的想法下,但对已经 UTF-8 进行编码也会使其更长。 So I can't detect latin1 vs UTF-8 that way.所以我无法以这种方式检测 latin1 与 UTF-8。
Question题
I would like to end up with always UTF-8 string, but how can I detect if it is latin1 or UTF-8, so I only convert the latin1 string?我想最终得到 UTF-8 字符串,但是我如何检测它是 latin1 还是 UTF-8,所以我只转换 latin1 字符串?
Being able to get a yes/no if a string is UTF-8 would be just as useful.如果字符串是 UTF-8,能够得到是/否也同样有用。
Due to some properties of UTF-8, it's very unlikely that text encoded using iso-8859-1 would be valid UTF-8 unless it decodes identically using both encodings [1] .由于 UTF-8 的某些属性,使用 iso-8859-1 编码的文本不太可能是有效的 UTF-8,除非它使用两种编码[1] 进行相同的解码。
As such, the solution is to try decoding it using UTF-8.因此,解决方案是尝试使用 UTF-8 对其进行解码。 If it fails, decode it using iso-8859-1 instead.如果失败,请改用 iso-8859-1 对其进行解码。 Since decoding using iso-8859-1 is a no-op, I'll be skipping that step.由于使用 iso-8859-1 进行解码是无操作的,因此我将跳过该步骤。
utf8:: implementation: utf8:: 实现:
my $decoded_text = $utf8_or_latin1; utf8::decode($decoded_text);
Encode:: implementation:编码::实现:
use Encode qw( decode_utf8 ); my $decoded_text = eval { decode_utf8($utf8_or_latin1, Encode::FB_CROAK|Encode::LEAVE_SRC) } // $utf8_or_latin1;
Now, you say you want UTF-8.现在,您说您想要 UTF-8。 UTF-8 is obtained from encoding decoded text. UTF-8 是从编码解码文本中获得的。
utf8:: implementation: utf8:: 实现:
my $utf8 = $decoded_text; utf8::encode($utf8);
Encode:: implementation:编码::实现:
use Encode qw( encode_utf8 ); my $utf8 = encode_utf8($decoded_text);
Notes笔记
Assuming the text is either valid UTF-8 or valid iso-8859-1, my solution would only guess wrong if all of the following are true:假设文本是有效的 UTF-8 或有效的 iso-8859-1,如果以下所有内容都为真,我的解决方案只会猜测错误:
(<80>..<9F> are unassigned or unprintable control characters, not sure which.) (<80>..<9F> 是未分配或不可打印的控制字符,不确定是哪个。)
In other words, that code is very reliable.换句话说,该代码非常可靠。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.