Get tokens from a string and change the string

Question

I want to separate a string into smaller ones and change the input string along the way.

So for exemple -> Input: Hi my name is

I want to get the first token (Hi), return it from a function and inside this same function change my string to: my name is

Then, next iteration the function returns "my" and the string becomes "name is"

Something like:

char * tokeniz(char * str){
    char * token;

    token = strtok(str," ");

    /* eliminate the token from the input string */

    return token;
}

main(){

    char *tok;
    char *s=malloc(sizeof(char)*100);

    fgets (s, 100, stdin);

    loop {
        tok=tokeniz(s);
        func_do_something_with_tok();
    }
}

Notice the input string 's' must be changed in the main function when returning for the next iteration

Answer 1

This is an interesting request: breaking a string in place into tokens without allocation or static state.

Here is a method:

• You can scan the argument string to determine the substring to return and the new start of the argument string.
• If no new token, return NULL
• Copy the token to a temporary block of memory
• Move the contents of the string in place for the new start to appear at the start.
• Copy the saved token after the new end of the string
• Return a pointer to that.
• One caveat: the source string must have at least one extra byte available in addition to the null terminator.

It is not as efficient as strtok_r because of the byte shuffling.

Here is an implementation:

#include <string.h>

char *tokeniz(char *str) {
    size_t n1 = strspn(str, " \f\t\r\n");
    size_t n2 = strcspn(str + n1, " \f\t\r\n");
    size_t n3 = strlen(str + n1 + n2) + 1;
    if (n2 == 0)
        return NULL;
    char token[n2];
    memcpy(token, str + n1, n2);
    memmove(str, str + n1 + n2, n3);
    char *res = str + n3;
    memcpy(res, token, n2);
    res[n2] = '\0';
    return res;
}