Same regex but giving different result with StringTokenizer and Scanner class delimiter

Question

Im trying to separate each word in the sentence using StringTokenizer class. It works fine for me. But I found another solution to my case using Scanner class.I applied same regular expression in both ways but got different result. I would like to know the reason for different out put I got but with same expression.
Here is my code :

String sentence = "I have some problems with this section!"
        + " But I love to learn.";


StringTokenizer st = new StringTokenizer(sentence, "[! ]");
System.out.println("========Using StringTokenizer=========");
while (st.hasMoreTokens()) {
    System.out.println(st.nextToken());
}


Scanner s = new Scanner(sentence);
s.useDelimiter("[! ]");
System.out.println("========Using Delimiter=========");
while (s.hasNext()) {
    System.out.println(s.next());

}

Out-put form StringTokenizer:

========Using StringTokenizer=========
I
have
some
problems
with
this
section
But
I
love
to
learn.

Out-put using Scanner class :

========Using Delimiter=========
I
have
some
problems
with
this
section

But
I
love
to
learn.

Answer 1

It is because Scanner may match an empty String , while StringTokonizer will not. In this case in the part "section! But" Scanner matches the whitespace after the ! symbol, whereas StringTokenizer does not.

Answer 2

Scanner includes empty matches while StringTokenizer does not.

StringTokenizer can't properly parse delimited files with meaningful indexed columns/fields like /etc/passwd or CSVs for this reason because it will not return all of the columns/fields while Scanner will.