Lexical Analyzer C程序，用于识别令牌

Question

I wrote a C program for lex analyzer (a small code) that will identify keywords, identifiers and constants. I am taking a string (C source code as a string) and then converting splitting it into words.

#include <stdio.h>
#include <conio.h>
#include <string.h>

char symTable[5][7] = { "int", "void", "float", "char", "string" };

int main() {
    int i, j, k = 0, flag = 0;
    char string[7];
    char str[] = "int main(){printf(\"Hello\");return 0;}";
    char *ptr;
    printf("Splitting string \"%s\" into tokens:\n", str);
    ptr = strtok(str, " (){};""");
    printf("\n\n");
    while (ptr != NULL) {
        printf ("%s\n", ptr);

        for (i = k; i < 5; i++) {
            memset(&string[0], 0, sizeof(string));
            for (j = 0; j < 7; j++) {
                string[j] = symTable[i][j];
            }

            if (strcmp(ptr, string) == 0) {
                printf("Keyword\n\n");
                break;
            } else
            if (string[j] == 0 || string[j] == 1 || string[j] == 2 ||
                string[j] == 3 || string[j] == 4 || string[j] == 5 ||
                string[j] == 6 || string[j] == 7 || string[j] == 8 ||
                string[j] == 9) {
                printf("Constant\n\n");
                break;
            } else {
                printf("Identifier\n\n");
                break;
            }
        }
        ptr = strtok(NULL, " (){};""");
        k++;
    }
    _getch();
    return 0;
}

With the above code, I am able to identify keywords and identifiers but I couldn't obtain the result for numbers. I've tried using strspn() but of no avail. I even replaced 0,1,2...,9 to '0','1',....,'9' .

Any help would be appreciated.

Answer 1

Here are some problems in your parser:

• The test string[j] == 0 does not test if string[j] is the digit 0 . The characters for digits are written '0' through '9' , their values are 48 to 57 in ASCII and UTF-8. Furthermore, you should be comparing *p instead of string[j] to test if you have a digit in the string indicating the start of a number.

• Splitting the string with strtok() is not a good idea: it modifies the string and overwrites the first separator character with '\\0' : this will prevent matching operators such as ( , ) ...

• The string " (){};""" is exactly the same as " (){};" . In order to escape " inside strings, you must use \\" .

To write a lexer for C, you should switch on the first character and check the following characters depending on the value of the first character:

• if you have white space, skip it
• if you have // , it is a line comment: skip all characters up to the newline.
• if you have /* , it is a block comment: skip all characters until you get the pair */ .
• if you have a ' , you have a character constant: parse the characters, handling escape sequences until you get a closing ' .
• if you have a " , you have astring literal. do the same as for character constants.
• if you have a digit, consume all subsequent digits, you have an integer. Parsing the full number syntax requires much more code: leave that for later.
• if you have a letter or an underscore: consume all subsequent letters, digits and underscores, then compare the word with the set of predefined keywords. You have either a keyword or an identifier.
• otherwise, you have an operator: check if the next characters are part of a 2 or 3 character operator, such as == and >>= .

That's about it for a simple C parser. The full syntax requires more work, but you will get there one step at a time.

Answer 2

When you're writing lexer, always create specific function that finds your tokens (name yylex is used for tool System Lex , that is why I used that name). Writing lexer in main is not smart idea, especially if you want to do syntax, semantic analysis later on.

From your question it is not clear whether you just want to figure out what are number tokens, or whether you want token + fetch number value. I will assume first one.

This is example code , that finds whole numbers:

int yylex(){

    /* We read one char from standard input */
    char c = getchar();

    /* If we read new line, we will return end of input token */
    if(c == '\n')
        return EOI;

    /* If we see digit on input, we can not return number token at the moment. 
         For example input could be 123a and that is lexical error  */
    if(isdigit(c)){

        while(isdigit(c = getchar()))
            ;

        ungetc(c,stdin);
        return NUM;
    }

    /* Additional code for keywords, identifiers, errors, etc. */
}

Tokens EOI , NUM , etc. should be defined on top. Later on, when you want to write syntax analysis, you use these tokens to figure out whether code responds to language syntax or not. In lexical analysis, usually ASCII values are not defined at all, your lexer function would simply return ')' for example. Knowing that, tokens should be defined above 255 value. For example:

#define EOI 256
#define NUM 257

If you have any futher questions, feel free to ask.

Answer 3

string[j]==1

This test is wrong ⁽¹⁾ (on all C implementations I heard of), since string[j] is some char eg using ASCII (or UTF-8 , or even the old EBCDIC used on IBM mainframes) encoding and the encoding of the char digit 1 is not the the number 1. On my Linux/x86-64 machine (and on most machines using ASCII or UTF-8, eg almost all of them) using UTF-8, the character 1 is encoded as the byte of code 48 (that is (char)48 == '1' )

You probably want

string[j]=='1'

and you should consider using the standard isdigit (and related) function.

Be aware that UTF-8 is practically used everywhere but is a multi-byte encoding (of displayable characters). See this answer .

---

Note (1): the string[j]==1 test is probably misplaced too! Perhaps you might test isdigit(*ptr) at some better place.

PS. Please take the habit of compiling with all warnings and debug info (eg with gcc -Wall -Wextra -g if using GCC ...) and use the debugger (eg gdb ). You should have find out your bug in less time than it took you to get an answer here.