实施词汇分析器

Question

I have an assignment to implement a lexical analyzer for a language c--. We must translate the c-- code into a series of tokens that will be represented as integers internally, since it will be easier to manipulate. Some lexical conventions of the language are that there are keywords like double, else, if, int, return,void, and while. Also special symbols like + - * / < <= > >= == != = ; , . ( ) [ ] { } /* */ //. identifiers can begin with any letter or underscore followed by any combination of letters, numbers and underscores. white spaces separate tokens and are ignored. numbers can be integers or decimals and comments lines and blocks are allowed.

import java.io.*;
public class Lex {

    public static boolean contains(char[] a, char b){
        for (int i = 0; i < a.length; i++) {
            if(b == a[i])
                return true;
        }
        return false;
    } 
    public static void main(String args[]) throws FileNotFoundException, IOException{

        //Declaring token values as constant integers.
        final int T_DOUBLE = 0; 
        final int T_ELSE = 1;
        final int T_IF = 2; 
        final int T_INT = 3;
        final int T_RETURN = 4; 
        final int T_VOID = 5;
        final int T_WHILE = 6; 
        final int T_PLUS = 7;
        final int T_MINUS = 8; 
        final int T_MULTIPLICATION = 9;
        final int T_DIVISION = 10; 
        final int T_LESS = 11;
        final int T_LESSEQUAL = 12; 
        final int T_GREATER = 13;
        final int T_GREATEREQUAL = 14; 
        final int T_EQUAL = 16;
        final int T_NOTEQUAL = 17;
        final int T_ASSIGNOP = 18; 
        final int T_SMEICOLON = 19;
        final int T_PERIOD = 20; 
        final int T_LEFTPAREN = 21;
        final int T_RIGHTPAREN = 22; 
        final int T_LEFTBRACKET = 23;
        final int T_RIGHTBRACKET = 24; 
        final int T_LEFTBRACE = 25;
        final int T_RIGHTBRACE = 26; 
        final int T_ID = 27;
        final int T_NUM = 28;
        char[] letters_ = {'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z','A','B','C','D',
            'E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','_'};
        char[] numbers = {'0','1','2','3','4','5','6','7','8','9'};
        char[] symbols = {'+','-','*','/','<','>','!','=',':',',','.','(',')','[',']','{','}'};
        FileInputStream fstream = new FileInputStream("src\\testCode.txt");
        DataInputStream in = new DataInputStream(fstream);
        BufferedReader br = new BufferedReader(new InputStreamReader(in));
        BufferedWriter bw1 = new BufferedWriter(new FileWriter(new File("src\\output.txt"), true));
        BufferedWriter bw2 = new BufferedWriter(new FileWriter(new File("src\\output2.txt"), true));
        String scanner;String temp = "";
        int n = 0;
        while((scanner = br.readLine()) != null){
            for (int i = 0; i < scanner.length(); i++) {
                for (int j = 0; j < scanner.length(); j++) {
                    if(contains(letters_,scanner.charAt(i)) || contains(numbers,scanner.charAt(i)) || contains(symbols,scanner.charAt(i))){
                        j++;
                        n++;
                        if(scanner.charAt(j) == ' ' || scanner.charAt(j) == '\n' || scanner.charAt(j) == '\t'){

                        }
                    }

                }

            }
        }

        in.close();


    }

}

This is our test code:

int fact(int x) {
// recursive factorial function 
   if (x>1) 
      return x * fact(x-1);
   else return 1;
}

void main(void) {
/* CS 311 project 2
A lexical analyzer */
   int x, y, z;
   double _funny;
   x = get_integer();
   _Funny = get_double();
   if (x>0) 
      print_line(fact(x));
   else if (_funny != 3.14) 
      print_line(x*_funny);
}

This should be our output

3 27 21 3 27 22 25 2 21 27 13 28 22 4 27 9 27 21 27 8 28 22 18 1 4 28 18 26 5 27 21 5 22 25 3 27 19 27 19 27 18 0 27 18 27 17 27 21 22 18 27 17 27 21 22 18 2 21 27 13 28 22 27 21 27 21 27 22 22 18 1 2 21 27 12 28 22 27 21 27 9 27 22 18 26

INT id leftparen INT id rightparen leftbrace IF leftparen id greater num rightparen RETURN id multiplication id leftparen id minus num rightparen semicolon ELSE RETURN num semicolon rightbrace VOID id leftparen VOID rightparen leftbrace INT id comma id comma id semicolon DOUBLE id semicolon id assignop id leftparen rightparen semicolon id assignop id leftparen rightparen semicolon IF leftparen id greater num rightparen id leftparen id leftparen id rightparen rightparen semicolon ELSE IF leftparen id notequal num rightparen id leftparen id multiplication id rightparen semicolon rightbrace

Ok ive written some code based on user John's suggestion. Im still confused on how this will work. When i iterate the second loop to find white space or a symbol how I know what type of token came before the ws of symbol. I've tried to put chracters i skip in a string and use a case statement to determine it but I think it writes the whole file into the string so my tokens never match. Also how can method find comments and safely ignore them?

Answer 1

There are a couple of different ways to approach this program. Without writing the code, I will try to explain what you need to do.

From the example you have submitted.

Your instructor has given you the key to the program. He has given you the output and you can construct a state table.

You can either go through the output and manually do this to check your answer, or create a small program to do this for you.

This is a table with the state number on the left, and the corresponding word on the right.

         3  int,  
         27 ID,
         21 leftparen, 
         22 right paren,
         25 left brace s, 
         2  if,
         13  greater, 

and so on.

You will need to create an input buffer
2 output buffers
2 loops one outer and one inner loop
1 case statement that corresponds to the state table.

when your going through the input buffer, you initialize the outer loop initialize the inner loop compare this first character and determine if its a valid character? if not increment loops till you find a valid character

Once you find the valid character it is the beginning of a token. Then find the end of the token by incrementing the inner loop by finding white space, or a special symbol. Then use a case statement to output the number in one buffer and the word that corresponds to the second buffer.

Then print out the number buffer. Then print out the word buffer.

Then increment outer loop to inner loop + 1 Make inner loop equal to outer loop

continue till you find the End of File. If they match your teachers output you are finished. If not you have a logic error. Then check to which value is invalid, and look at that part of the program.

Its been 20 yrs guys.

Answer 2

Pretty familiar tast, except for the fact I was writing LLK analizer... In your case, try to look in the way of formal grammars and like - you're step is almost required step before performing analisys by those grammars. Maybe some working parsers (opensource) like lex && flex would help.

INHO, the easiest way is to read input file character by character into some string and check, does this string matches one of your regexps... If it does - write appropriate code to the output and clear the string you're using as buffer. There are two problems in this case: this works in O(n*m), where n is the length of you're text and m is number of regular expressions you have (in worth case), and second - you must not use prefixed expressions...I meen that you must not have any expression to have another one as prefix (beginning), or this expression would be unreachable.