Struggle with function strtok() in C

Question

I need to use strtok function to analyze every word in some strings. I wrote a code like:

char *token;
token=strtok(string,symbol);
while(token!=NULL){
   functionX(token); //this is a function that anlayze the token
   token=strtok(NULL,symbol);
}

but the "functionX" only receive first words of the strings and null pointer then. If I put

printf("%s",token);

instead of functionX it print every piece of the string. How can I solve this?

This is what I called "functionX":

void match(char *token, char *code){
FILE *rules;
char *tmp_token;
char stream[15];
int found;
rules=fopen("rules.vk","r");
found=0;
while((fgets(stream,sizeof(stream),rules))>0){
    tmp_token=strtok(stream,";");
    if((strcmp(tmp_token,token))==0){
        strcpy(code,strtok(NULL,";"));
        found=1;
    }
}
if(found==0) strcpy(code,token);

}

Answer 1

This is one of the difficulties in using strtok . It has internal state in the routine, which keeps track of the position in the string that was passed in originally (ie the first strtok(string, symbol); call).

This information gets messed up when you call strtok inside the functionX as it changes the internal pointer. Then when you return you're using this erroneous state.

What you need to use is the strtok_r routine, which keeps a private copy of this pointer which you have to pass into the call to strtok_r .

As an example for the original routine, you could change it to:

char *token;
char *save;
token=strtok_r(string,symbol, &save);
while(token!=NULL){
   functionX(token); //this is a function that anlayze the token
   token=strtok_r(NULL,symbol, &save);
}

and the inner routine could be changed to:

void match(char *token, char *code){
    FILE *rules;
    char *tmp_token;
    char *save;
    char stream[15];
    int found;
    rules=fopen("rules.vk","r");
    found=0;
    while((fgets(stream,sizeof(stream),rules))>0){
        tmp_token=strtok_r(stream,";", &save);
        if((strcmp(tmp_token,token))==0){
            strcpy(code,strtok_r(NULL,";", &save));
            found=1;
        }
    }
    if(found==0) strcpy(code,token);
}