Strtok_r segmentation fault

Question

I know this has been asked before, but previous answers have said you can't modify a string literal, and unless I am misunderstanding I am not. I can sucesfully use strtok_r until the very last line of input, then I get a seg fault. Why would this happen? Heres a simplified version of my code. From printing I know it processes through get b, but then there is a seg fault?

#include <stdio.h>
#include <string.h>   //strlen
#include <stdlib.h>

    void handler(char * input){
    char * end_str;
    char * token =strtok_r(input, "\n", &end_str);
    char * endd_str;
    char * token2= strtok_r(token, " ", &endd_str);
    while(token!=0){
        printf("%s\n",token);
        token=strtok_r(NULL,"\n",&end_str);
        while(token2 !=0){
            token2 =strtok_r(NULL," ",&endd_str);
            printf("%s\n",token2);}
    }

}

int main(int argc , char *argv[])
{
    char test[100]="set a,10.5\nset b,11.5\nget b\nadd e,a,b\nsub g,a,b";
    handler(test);
    return 0;
}

Answer 1

You're interleaving strtok_r calls working off of the same string. On the second iteration of the inner while loop, before the call to token2 =strtok_r(NULL," ",&endd_str); , endd_str points to an empty string because the string it was matching against reached the end, so strtok_r returns NULL . Passing that NULL to printf causes the segfault.

After each parse of the "\\n" delimiter, do a fresh start of looping through the " " delimeter. Don't do the outer one again until you've exhausted the inner one. Also, move your strtok_r calls to the end of the loop so the while condition checks them properly.

You need to do the following:

void handler(char * input){
    char * end_str;
    char * token =strtok_r(input, "\n", &end_str);
    while(token!=0){
        printf("%s\n",token);
        char * endd_str;
        char * token2= strtok_r(token, " ", &endd_str);
        while(token2 !=0){
            printf("%s\n",token2);
            token2 =strtok_r(NULL," ",&endd_str);    // call at end of loop
        }
        token=strtok_r(NULL,"\n",&end_str);          // call at end of loop
    }
}

Ouput:

set a,10.5
set
a,10.5
set b,11.5
set
b,11.5
get b
get
b
add e,a,b
add
e,a,b
sub g,a,b
sub
g,a,b

Answer 2

Works perfectly for me (MS Visual C++). Only difference is usage strtok_s instead of strtok_r considering it is Windows instead of POSIX.

Here I have added some modifications to your code.

#include <string.h>   //strlen
#include <stdlib.h>

void handler(char * input){
    char * end_str;
    char * token =strtok_s(input, "\n", &end_str); // tokenize input string
    char * endd_str;
    char * token2 = NULL;

    while(token!=0){

        printf("%s\n",token);

        token2 = strtok_s(token, " ", &endd_str); // retokenize first token
        while(token2 !=0){
            printf("%s\n",token2); // replace printf call to avoid null string output and first token miss
            token2 =strtok_s(NULL," ",&endd_str);}

        token = end_str; // "important" restore char pointer to point the last untokenized input character

        token=strtok_s(token,"\n",&end_str); // the same as printf issue is for all loops body
    }

}

int main(int argc , char *argv[])
{
    char test[100]="set a,10.5\nset b,11.5\nget b\nadd e,a,b\nsub g,a,b";
    handler(test);
    return 0;
}