java无法从文件中读取一行

Question

I'm reading a file with the following piece of code:

 Scanner in = new Scanner(new File(fileName));
    while (in.hasNextLine()) {
        String[] line = in.nextLine().trim().split("[ \t]");
       .
       .
       .
    }

When I open the file with the vim, some lines begin with the following special character:

but the java code can't read these lines. When it reaches these lines it thinks that it's the end of the file and hasNextLine() function returns false!!

EDIT: this is the hex dump of the mentioned (problematic) line:

0000000: e280 9c20 302e 3230 3133 3220 302e 3231 ... 0.20132 0.21 0000010: 3431 392d 302e 3034 0a 419-0.04.

Answer 1

@VGR got it right.

tl;dr: Use Scanner in = new Scanner(new File(fileName), "ISO-8859-1");

What appears to be happening is that:

• Your file is not valid UTF-8 due to that lone 0x9C character.
• The Scanner is reading the file as UTF-8 since this is the system default
• The underlying libraries throw a MalformedInputException
• The Scanner catches and hides it (a well meaning but misguided design decision)
• It starts reporting that it has no more lines
• You won't know anything's gone wrong unless you actually ask the Scanner

Here's a MCVE:

import java.io.*;
import java.util.*;

class Test {
  public static void main(String[] args) throws Exception {
    Scanner in = new Scanner(new File(args[0]), args[1]);
    while (in.hasNextLine()) {
      String line = in.nextLine();
      System.out.println("Line: " + line);
    }
    System.out.println("Exception if any: " + in.ioException());
  }
}

Here's an example of a normal invocation:

$ printf 'Hello\nWorld\n' > myfile && java Test myfile UTF-8
Line: Hello
Line: World
Exception if any: null

Here's what you're seeing (except that you don't retrieve and show the hidden exception). Notice in particular that no lines are shown:

$ printf 'Hello\nWorld \234\n' > myfile && java Test myfile UTF-8
Exception if any: java.nio.charset.MalformedInputException: Input length = 1

And here it is when decoded as ISO-8859-1, a decoding in which all byte sequences are valid (even though 0x9C has no assigned character and therefore doesn't show up in a terminal):

$ printf 'Hello\nWorld \234\n' > myfile && java Test myfile ISO-8859-1
Line: Hello
Line: World
Exception if any: null

If you're only interested in ASCII data and don't have any UTF-8 strings, you can simply ask the scanner to use ISO-8859-1 by passing it as a second parameter to the Scanner constructor:

Scanner in = new Scanner(new File(fileName), "ISO-8859-1");