C词法分析器。 使用开关来分析和计算小数/非小数

Question

My lexical analyzer recognizes digits(5,555,543667), decimals(44.65,4.1), and periods(.).

I can count digits, decimals, and periods fine but when I come across a digit and period next to each other it counts it as a decimal.

Consider a text file that contains: 555 2.3 55.23 44 5.

My output would be

1 type 1: 555
2 type 3: 2.3
3 type 3: 55.23
4 type 1: 44
5 type 3: 5.

Where type 3 is my identifier for a decimal.

I would want the 5th and 6th tokens to be counted as a digit and then a period.

Here is how I am handling my switch statement.

  switch(*b) {

    case '0':
    case '1':
    case '2':
    case '3':
    case '4':
    case '5':
    case '6':
    case '7':
    case '8':
    case '9':
    digits:
        t.length++;
        switch(*(b + t.length)) {
            case '0':
            case '1':
            case '2':
            case '3':
            case '4':
            case '5':
            case '6':
            case '7':
            case '8':
            case '9':
                goto digits;
            case '.': 
                goto decimal;                   
                break;
            default:
                break;
        }

         t.type = TOKEN_DIGITS;
        t.string = (char *)calloc(t.length + 1, sizeof(char));
        strncpy(t.string, b, t.length);
        break;



    decimal:
        t.length++;
        switch(*(b + t.length)) {
            case '0':
            case '1':
            case '2':
            case '3':
            case '4':
            case '5':
            case '6':
            case '7':
            case '8':
            case '9':
                goto decimal;
                break;
            }   
            t.type = TOKEN_DECIMAL;
            t.string = (char *)calloc(t.length+1,sizeof(char));
            strncpy(t.string,b,t.length);           
       break;

Tried multiple things but I am officially stuck.

Answer 1

You really should be using character classification functions for this kind of excercise instead of long switch statements. Your code will be a lot simpler and you won't have to use goto at all.

For example, a number could be described with the following regular expression (added whitespace to break up the various blocks):

[-+]? [0-9]* \.? [0-9]+

This already shows the possible state transitions:

• A number can (optionally) start with + or - (if you support signed numbers)
• It may have 0..n digits
• If the following character is not a decimal point symbol, it should be a separator, otherwise it's an invalid symbol. If it's a separator, your number is terminated.
• After a decimal point there should be 1..n digits
• The number is terminated when you reach the end of input or you encounter a separator

All this can be done in a handful of lines of code - just have a pointer that points to your current input character and then keep stepping forward one by one and examine each character and based on the character class, decide what to do.

Now, this particular approach doesn't handle floating point numbers using scientific notation, etc. but adding thos extras is really simple once you have the basics done.

Answer 2

I think this complements xxbbcc's answer.

* Very roughly * something like this.

#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>

yylex() {
        int c;
        char *p, buf[1000];

        for(c = get(); isspace(c); c = get());

        if(isdigit(c)) {
                p = buf;
                while(isdigit(c)) {
                        *p++ = c;
                        c = get();

                }
                *p = 0;
                if(c != '.') {
                        unget(c);
                        int i = atoi(buf);
                        return INT;
                }
                assert(c == '.');
                *p++ = c;
                c = get();
                while(isdigit(c)) {
                        *p++ = c;
                        c = get();
                }
                *p = 0;
                float f = atof(buf);
                unget(c);
                return DECIMAL;
        }
}

There's a lot of details left unsaid. Watching for EOF. Buffer overflow. Setting yylval to the int or float. Parsing tokens other than simple numbers.

Answer 3

Use a variable like digit_follow_peroid to keep the state. Every time you encounter a peroid, set the variable to false and then when you encounter a digit in the decimal switch block, set it to true. check the variable's value to determine the t.lengthbefore strncpy . Maybe you also need other variables to work together with it. The best way is to define a state-transition matrix, which is much better than gotos.