C-使用strtok的嵌套循环

Question

I am trying to use strtok to split up a text file into strings that I can pass to a spell check function, the text file includes characters such as '\\n', ' ?!,.' etc... I need to print any words that fail the spell check and the line number that they are on. Keeping track of the line is what I'm struggling with. I have tried this so far but it only returns results for the first line of the text file:

char str[409377];
fread(str, noOfChars, 1, file);
fclose(file);

int lines=1;
char *token;
char *line;
char splitLine[] = "\n";
char delimiters[] = " ,.?!(){}*&^%$£_-+=";
line = strtok(str, splitLine);
while(line!=NULL){
    token = strtok(line, delimiters);
    while(token != NULL){
        //print is just to test if I can loop through all the words
        printf("%s", token);
        //spellCheck function & logic here
        token = strtok(NULL, delimiters);
    }
    line = strtok(NULL, splitLine);
    lines++
}

Is using the nested while loop and strtok possible? Is there a better way to keep track of the line number?

Answer 1

The strtok function is not reentrant ! It can not be used to tokenize multiple strings simultaneously. It's because it keeps internal state about the string currently being tokenized.

If you have a modern compiler and standard library then you could use strtok_s instead. Otherwise you have to come up with another solution.

Answer 2

You can use strtok, but it's not very easy to use. It's a stupid function, all it really does is replace delimiters with nuls and return a pointer to the start of the sequence it has delimited. So it's destructive. It can't handle special cases like English words being allowed one apostrophe (we're is a word, we'r'e is not), you have to make sure you list all the delimiters specifically.

It's probably best to write mystrok yourself, so you understand how it works. Then use that as the basis for your own word extractor.

The reason for your bug is that you chop off the first line, then that is all that strok sees on the subsequent calls.