编码问题； .jar不适用于UTF-8文件中的西里尔字母

Question

So I have this regex as String literal in my code:

private static final String FILE_PATTERN = "((\\s*\".*НЕКОТОРЫЕ СИМВОЛЫ .*\"\\R)([^\"].* (?!-)\\d+\\s*)+)+";

Also I have input test files in UTF-8 encoding.

And the problem is that when I test my program in IDE (IntelliJ IDEA in my case) everything is OK. Particularly, regex works with Cyrillic characters in test files.

But when I build my program (Maven) and tested .jar file with the same test files, it turned out that most likely regex won't work with Cyrillic characters.

Then I tested it again with file in Windows 1251 encoding and it worked.

So my question is - how can I make my .jar work with UTF-8 files, just like in IDE?

Thanks in advance.

---

[UPDATE1]

two test files, one in UTF-8 and another in Windows 1251

I've tried to replace Cyrillic characters with \\u codes like this:

private static final String FILE_PATTERN = "((\\s*\".*\\u041E\\u0442\\u0434\\u0435\\u043B .*\"\\R)([^\"].* (?!-)\\d+\\s*)+)+";

this doesn't work :(

---

[UPDATE2]

File processing starts like this:

static void processFile(String inputFile) {
    try {
        String fileStr = FileHandler.readFile(inputFile).toString();
        if (!FileParser.validateFile(fileStr)) {
            System.out.println("Sorry, input file format is invalid");
            ...

File validating looks like this:

public class FileParser {
private static final String FILE_PATTERN = "((\\s*\".*Отдел .*\"\\R)([^\"].* (?!-)\\d+\\s*)+)+";

public static boolean validateFile(String fileStr) {
    return Pattern.compile(FILE_PATTERN).matcher(fileStr).matches();
}
...

File reading is very common I think:

public class FileHandler {
public static StringBuilder readFile(String fileName) {
    StringBuilder res = new StringBuilder();
    String temp;
    try (BufferedReader r = new BufferedReader(new FileReader((fileName)))) {
        while ((temp = r.readLine()) != null) {
            res.append(temp).append("\n");
        }
    } catch (FileNotFoundException e) { 
        System.out.println("Input file not found!");
    } catch (IOException e) {
        // log exception
    }
    return res;
}
...

Answer 1

I'll throw some possibilities at the problem.

The classes FileReader and FileWriter use the default platform encoding, without overload for a specified encoding. I am not sure whether this is intended, but one of the alternatives:

public static StringBuilder readFile(String fileName) {
    StringBuilder res = new StringBuilder();
    String temp;
    Charset charset = StandardCharsets.UTF_8;
    //Charset charset = Charset.fromName("Windows-1251");
    try (BufferedReader r = Files.newBufferedReader(fileName, charset)) {
        while ((temp = r.readLine()) != null) {
            res.append(temp).append("\n");
        }
    } catch (FileNotFoundException e) { 
        System.out.println("Input file not found!");
    } catch (IOException e) {
        // log exception
    }
    return res;
}

Or:

String readFile(String fileName) throws IOException {
    byte[] content = Files.readAllBytes(Paths.get(fileName));
    return new String(content, StandardCharsets.UTF_8);
}

Then the editor encoding of the java sources must be the same encoding as that of the javac compiler . One can check this by using the \\uXXXX ASCII representation of such special chars: if it then suddenly works, ...

You used two backslashes, but \c (letter c ) works java source level, and in fact instead of public class you can write publi\c \class .

private static final String FILE_PATTERN =
    "((\\s*\".*\u041E\u0442\u0434\u0435\u043B .*\"\\R)([^\"].* (?!-)\\d+\\s*)+)+";

Then there is the regular expression, that has two Unicode flags, (?u) and (?U) undermore for what a letter constitutes. That should not be a problem here.