Why the strtok function ignores two tokens one after the other

Question

I have a code that split my char str[] to tokens:

void my_function(char str[])
{   
    int i = 0;
    char *p = strtok(str, "/");
    char *rows[SIZE] = { NULL };

        while(p==NULL)
        {
            rows[i++] = p;
            p = strtok(NULL, "/");
        }
}

There is a problem with 2 tokens('/') one after other to save the orders of tokens

example: "abc/a//bb"

(SIZE=4)

rows=['abc','a','bb',null]

and I want that the result will be:

rows=['abc','a',null,'bb'] (save the orders of tokens)

How can I over this problem?

Answer 1

Yes, that is strtok() 's behavior. The tokens passed to strtok() are usually separated by some kind of whitespace, and that's the user's expectation when the separator is whitespace.

To overcome, implement your own tokenizer.

char *tokenize(char *s, char t, char **ctx)
{
    if (s) *ctx = s;
    if (!*ctx) return NULL;
    s = *ctx;
    if (!*s) return NULL;
    while (**ctx) {
       if (**ctx == t) {
           **ctx = 0;
           ++*ctx;
           return NULL;
       }
       ++*ctx;
    }
    return s;
}

Answer 2

strtok gives you tokens separated by strings of 1 or more delimiter characters, so by definition, you can't get empty tokens. If you want tokens separated by single character delimiters, use strsep instead. Unofrtunately, that is only available on BSD and Linux systems, but is easy enough to write yourself:

char *strsep(char **stringp, const char *delim) {
    char *rv = *stringp;
    if (rv) {
        *stringp += strcspn(rv, delim);
        if (**stringp)
            *(*stringp)++ = '\0';
        else
            *stringp = NULL;
    }
    return rv;
}