[英]Running Ascii regex over non-ASCII characters with UTF-8
#include <boost/regex.hpp>
#include <string>
#include <vector>
#include <iostream>
int main(int argc, char* argv[]) {
std::string text = argv[1];
std::string patterns = argv[2];
boost::regex regex = boost::regex(patterns);
boost::smatch match;
std::cout << boost::regex_search(text, match, regex) << std::endl;
}
If I run the program over the input hello¿ ¿
(containing a non-ASCII character with UTF-8 encoding) it returns 0
ie not found, but if I run it over the input hel√ √ (again containing non-ascii) it returns 1, ie found. 如果我通过输入
hello¿ ¿
(包含UTF-8编码的非ASCII字符)运行程序,它返回0
即未找到,但是如果我通过输入hel√√运行它(再次包含非ascii)它返回1,即找到。
My question: What is the expected behavior of boost::regex
(ie the ascii version) when run over utf characters? 我的问题:当运行utf字符时,
boost::regex
(即ascii版本)的预期行为是什么?
Edit: Thanks for all the comments, I am still interested as to why exactly 1 is output, since both the text and the regex contain non-ascii characters. 编辑:感谢所有的评论,我仍然感兴趣的是为什么输出正好1,因为文本和正则表达式都包含非ascii字符。 My guess would be that the bytes are interpreted as ascii and thus they match.
我的猜测是字节被解释为ascii,因此它们匹配。
Using regular expressions on ASCII strings, is about using "bytes" to find a pattern in. 在ASCII字符串上使用正则表达式,就是使用“bytes”来查找模式。
Using regular expressions on UTF-8 strings, is about using regular expressions on "multi-byte" sequences, where a sequence represents a Unicode code point. 在UTF-8字符串上使用正则表达式是关于在“多字节”序列上使用正则表达式,其中序列表示Unicode代码点。
Thus the regular expression gets applied to a Unicode string with an encoding with variable byte-count per character. 因此,正则表达式应用于Unicode字符串,其编码具有每个字符可变的字节数。
UTF-8 strings contain multi-byte sequences with 1 to 4 bytes, which representing a Unicode "character". UTF-8字符串包含1到4个字节的多字节序列,表示Unicode“字符”。 In UTF-8 only ASCII 7 bit characters are 1 byte "wide".
在UTF-8中,只有ASCII 7位字符是1字节“宽”。
So - using an ASCII regular expression engine on an UTF-8 encoded string , ignores the multi-byte sequences in the UTF-8 encoded string and causes a pattern matching byte by byte. 因此 - 在UTF-8编码的字符串上使用ASCII正则表达式引擎 ,忽略UTF-8编码字符串中的多字节序列,并导致逐字节匹配模式。 The results of this ASCII regular expression engine usage on an UTF-8 encoded string is invalid .
在UTF-8编码的字符串上使用此ASCII正则表达式引擎的结果无效 。
Please take a look at http://utfcpp.sourceforge.net . 请查看http://utfcpp.sourceforge.net 。
To get the regular expressions working on UTF-8 encoded strings, you have to … 要使正则表达式处理UTF-8编码的字符串,您必须...
std::codecvt_utf8
in combination of setting temporarily the global locale to get the regular expression working, or std::codecvt_utf8
结合临时设置全局语言环境以使正则表达式工作,或者 The regex_search function returns a boolean and true
on a match. regex_search函数在匹配时返回布尔值和
true
。
In your case the ASCII regular expression pattern matches a part of the UTF-8 encoded string, which is parsed invalidly as ASCII string - as you assumed! 在您的情况下,ASCII正则表达式模式匹配UTF-8编码字符串的一部分,该字符串被无效地解析为ASCII字符串 - 正如您所假设的那样!
If you have English text in an UTF-8 encoded string, then an ASCII regular expression engine can be used safely. 如果您使用UTF-8编码的字符串中的英文文本,则可以安全地使用ASCII正则表达式引擎。 Leaving the ASCII 7 bit range, makes the result of the ASCII regular expression engine unreliable.
保留ASCII 7位范围,使得ASCII正则表达式引擎的结果不可靠。
It is a bug and not a feature: I tried your example on a better system (g++ 4.9.2 on Windows MinGW) and all happens well: 这是一个错误,而不是一个功能:我在一个更好的系统上尝试了你的例子(在Windows MinGW上的g ++ 4.9.2),一切都很顺利:
#include <iostream>
#include <string>
#include <regex>
int main()
{ std::string text ="hello¿"; // or "hello√"
std::string patterns ="¿"; // or "√"
std::regex regex = std::regex(patterns);
std::smatch match;
std::cout << std::regex_search(text, match, regex) << std::endl;
}
with output: 输出:
1
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.