2字节UTF-8序列的无效字节2：如何查找字符

Question

I have a big text file on my windows machine in UTF-8 encoding. Somehow one or more of the characters in this file are invalid for UTF-8 encoding, giving error as "Invalid byte 2 of 2-byte UTF-8 sequence".

I am using windows 7, and I want to find the character which is invalid. I guess there is a UNIX command for this, but is there any tool or utility or regex(something which doesn't need to write a programe/code) which can be used in windows.

I can use notepad++ or PSPAD or similar text editor, or if there is any windows command, I can create a batch file. Please suggest.

Answer 1

Create a FileReader to read the file byte by byte. If the current byte looks like the first of a 2-byte UTF-8, read the next byte, put the two in a byte[2] array, and give this to new String(array, "UTF-8"). In the loop, count the bytes read, and catch the exception to produce the position and byte values.

Answer 2

It's possible that your UTF-8 file has Byte Order Mark on it, which is often not recognised by the Java Readers.

Open the file in Notepad++. If the file has a BOM, Notepad++ will report "UTF-8" rather than "UTF-8 w/o BOM".

You can either convert to UTF-8 without BOM or use something like: https://stackoverflow.com/a/2905038/1554386 to strip the BOM.