将字符串拆分为不同字符类型的子字符串列表

Question

I am writing a spell checker that takes a text file as input and outputs the file with spelling corrected.

The program should preserve formatting and punctuation.

I want to split the input text into a list of string tokens such that each token is either 1 or more: word, punctuation, whitespace, or digit characters .

For example:

Input:

words.txt:

asdf don't ]'.'..;'' as12....asdf.
asdf

Input as list:

["asdf" , " " , "don't" , " " , "]'.'..;''" , " " , "as" , "12" , "...." , "asdf" , "." , "\\n" , "asdf"]

Words like won't and i'll should be treated as a single token.

Having the data in this format would allow me to process the tokens like so:

String output = "";

for(String token : tokens) {
    if(isWord(token)) {
        if(!inDictionary(token)) {
            token = correctSpelling(token);
        }
    }
    output += token;
}

So my main question is how can i split a string of text into a list of substrings as described above? Thank you.

Answer 1

The main difficulty here would be to find the regex that matches what you consider to be a "word". For my example I consider ' to be part of a word if it's proceeded by a letter or if the following character is a letter:

public static void main(String[] args) {
        String in = "asdf don't ]'.'..;'' as12....asdf.\nasdf";

        //The pattern: 
        Pattern p = Pattern.compile("[\\p{Alpha}][\\p{Alpha}']*|'[\\p{Alpha}]+");

        Matcher m = p.matcher(in);
        //If you want to collect the words
        List<String> words = new ArrayList<String>();

        StringBuilder result = new StringBuilder();

        Now find something from the start
        int pos = 0; 
        while(m.find(pos)) {
            //Add everything from starting position to beginning of word
            result.append(in.substring(pos, m.start()));

            //Handle dictionary logig
            String token = m.group();
            words.add(token); //not used actually
            if(!inDictionary(token)) {
                token = correctSpelling(token);
            }
            //Add to result
            result.append(token);
            //Repeat from end position
            pos = m.end();
        }
        //Append remainder of input
        result.append(in.substring(pos));

        System.out.println("Result: " + result.toString());
    }

Answer 2

Because I like solving puzzles, I tried the following and I think it works fine:

public class MyTokenizer {
    private final String str;
    private int pos = 0;

    public MyTokenizer(String str) {
        this.str = str;
    }

    public boolean hasNext() {
        return pos < str.length();
    }

    public String next() {
        int type = getType(str.charAt(pos));
        StringBuilder sb = new StringBuilder();
        while(hasNext() && (str.charAt(pos) == '\'' || type == getType(str.charAt(pos)))) {
            sb.append(str.charAt(pos));
            pos++;
        }
        return sb.toString();
    }

    private int getType(char c) {
        String sc = Character.toString(c);
        if (sc.matches("\\d")) {
            return 0;
        }
        else if (sc.matches("\\w")) {
            return 1;
        }
        else if (sc.matches("\\s")) {
            return 2;
        }
        else if (sc.matches("\\p{Punct}")) {
            return 3;
        }
        else {
            return 4;
        }
    }

    public static void main(String... args) {
        MyTokenizer mt = new MyTokenizer("asdf don't ]'.'..;'' as12....asdf.\nasdf");
        while(mt.hasNext()) {
            System.out.println(mt.next());
        }
    }
}