如何从AC文件中找到令牌？

Question

I am trying to generate tokens from a C source file. I have split the C file into an array line and stored the words of the entire file in an array words .

The problem is with the strtok() function, which is splitting the line on whitespace characters. Because of this, I am not getting certain delimiters like parentheses and brackets because there is no whitespace between them and other tokens.

How do I determine which one is an identifier and which one is an operator?

Code so far:

int main()
{
    /* ... */

    char line[300][200];
    char delim[]=" \n\t";
    char *words[1000];
    char *token;

    while (fgets(&line[i][0], 100, fp1) != NULL)
    {
        token = strtok(&line[i][0], delim);

        while (token != NULL)
        {
            words[j++] = token;
            token = strtok(NULL, delim);
        }

        i++;
    }

    for(i = 0; i < 50; i++)
    {
        printf("%s\n", words[i]);
    }

    return 0;
}

Answer 1

This is a tricky question, something that needs probably more depth than a StackOverflow answer. I'll try, nonetheless.

Tokenizing the input is the first part of the compilation process . The objective is to simplify the task of the parser, which is going to make an abstract syntax tree with the contents of the file. How do we simplify this? We do recognize those tokens that have a special meaning, also identifiers, operators... C is indeed a tricky, complex language. Let's simplify the language to tokenize: we'll start with a typical calculator.

An input example would be:

( 4 +5)* 2

When syntax is free, you can add or skip spaces, so as you have already experimented, splitting by space is not an option.

The tokenized output for the example above would be: LPAR, LIT, OP, LIT, RPAR, OP, LIT. The meaning goes as follows:

LPAR: Left parenthesis
RPAR: Right parenthesis
LIT:  Literal (a number)
OP:   Operator (say: +, -, * and /).

The complete ouput would therefore be:

{ LPAR, LIT(4), OP('+'), LIT(5), RPAR, OP('*'), LIT(2) }

Your lexer basically has to advance in the input string, char by char, using a state machine. For example, when you read a number, you enter in the "input literal" state, in which only other numbers and '.' are allowed.

Now the parser has an easier task. If you feed it with the previous tokens, it does not have to skip spaces, or distinguish between a negative number and a minus operator, it can just advance in a list or array. It can behave following the type of the token, and some of them have associated data, as you can see.

This is only an introduction of the introduction, anyway. Information about the whole compilation process could fill a book. And there are actually many books devoted to this topic, such as the famous " Dragon book " from Aho, Sethi&Ullman. A more updated one is the " Tiger book ".

Finally, lexers are quite similar among each others, and it is therefore possible to find generic lexers out there. You can also even find the C grammar for that kind of tools.

Hope this (somehow) helps.